77 lines
5.4 KiB
Markdown
77 lines
5.4 KiB
Markdown
# Beating dependency hell with GNU Guix
|
|
|
|
`mobilizon-reshare`'s distribution process relies quite a bit upon GNU Guix. It's involved in our CI pipeline, and it builds the [OCI compliant](https://opencontainers.org/) container image available on Docker Hub. It provides us with [inspectable](https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point/), [bit-for-bit reproducible](https://reproducible-builds.org/) and [fully bootstrappable](https://bootstrappable.org) images which in turns allows for strong control on what code is actually bundled within the image and it should prevent entire classes of supply-chain attacks starting from the [Trusting Trust attack](https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf) up until the many recent [attacks](https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021) to many FOSS software registries.
|
|
|
|
To allow for interoperability with the Python ecosystem, we also ship a `pyproject.toml` that we handle with [Poetry](https://python-poetry.org/). The next paragraph will elaborate on the interactions between Poetry and Guix.
|
|
|
|
## Update the dependency graph of mobilizon-reshare
|
|
|
|
> **Beware!** - Dependency updates are better delivered to master as a single commit, to avoid confusing the CI.
|
|
|
|
### Python dependencies
|
|
|
|
We **must** keep Poetry and Guix version as much aligned as possible, to prevent unpredictable behavior. All the following content assumes this invariant.
|
|
|
|
Everything starts from `pyproject.toml`: usually your IDE warns you about outdated dependency, so let's assume you want to bump the version of a Python package. First keep in mind that Poetry's [tilde requirements](https://python-poetry.org/docs/dependency-specification/#tilde-requirements) are SemVer compatible but stricter than caret requirements so they should make matching Guix version easier. Then it's time to actually edit `pyproject.toml` and bump the version of a package.
|
|
|
|
To update Python dependencies and test your changes the steps are:
|
|
|
|
```shell
|
|
$ poetry update
|
|
$ guix time-machine -C channels-lock.scm -- build -f guix.scm
|
|
$ scripts/build_docker_image.sh
|
|
```
|
|
|
|
If these steps succeed you can safely commit your changes. If Guix fails you have to examine the output of the command that failed and figure out the problem. 99% of the times it'll be a version mismatch, as Guix's [`python-build-system`](https://guix.gnu.org/en/manual/devel/en/guix.html#index-python_002dbuild_002dsystem) has a `sanity-check` phase that'll try to instantiate the entry point generated by Poetry that, among other things, checks for runtime dependencies versions and errors out if it finds a mismatch between the version actually available in the runtime environment and the version defined in `pyproject.toml`.
|
|
|
|
You now have two alternatives:
|
|
|
|
1. You try to follow the next step about system dependencies. `channels-lock.scm` locks everything: as long as the Guix commit specified in that file does not change, `guix time-machine` will look for the exact same package graph. This means that every time we build the image we get the same exact dependencies we ask for, but this semantics is slightly different from Poetry's lock-file which instead tracks the **latest version** (within the constraints) available on Pypi. Having a more updated Guix version may allow for a more updated mapping of Pypi.
|
|
2. You find the package (or packages) responsible for the mismatch and try to manipulate it to follow Poetry's constraints. This requires some basic Scheme understanding but nothing complex. There are many ways a Guix package can be programmatically manipulated as it's just a structured Scheme record, you can start by looking into [package variants](https://guix.gnu.org/en/manual/devel/en/guix.html#Defining-Package-Variants) or also directly at the [channel code](https://github.com/fishinthecalculator/mobilizon-reshare-guix/tree/main/modules/mobilizon-reshare).
|
|
|
|
### System dependencies
|
|
|
|
Python's own dependencies are dependencies too! Guix freezes the whole dependency graph of an artifact with [channels specifications](https://guix.gnu.org/en/manual/devel/en/guix.html#Replicating-Guix) so to update "system" dependencies you need to follow these steps.
|
|
|
|
First let's update our Guix version to the latest commit:
|
|
|
|
```shell
|
|
$ guix pull
|
|
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
|
|
Authenticating channel 'guix', commits 9edb3f6 to d41c82b (162 new commits)...
|
|
Building from these channels:
|
|
guix https://git.savannah.gnu.org/git/guix.git d41c82b
|
|
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%\
|
|
|
|
[...]
|
|
|
|
building package cache...
|
|
building profile with 3 packages...
|
|
$
|
|
```
|
|
|
|
Channels specification define the Guix commit that should be used to fetch the right dependency graph, so what we want to do is replace the commit in `channels-lock.scm` with the one we just pulled:
|
|
|
|
```shell
|
|
$ guix describe
|
|
Generation 31 Mar 12 2022 12:35:00 (current)
|
|
guix d41c82b
|
|
repository URL: https://git.savannah.gnu.org/git/guix.git
|
|
branch: master
|
|
commit: d41c82b481fd0f5c7d45d6e2629fdf9d2085205b
|
|
|
|
$ vim channels-lock.scm
|
|
```
|
|
|
|
To test our change we can run:
|
|
|
|
```shell
|
|
$ guix time-machine -C channels-lock.scm -- build -f guix.scm
|
|
```
|
|
|
|
But a better test would be to build the Docker image, as that actually bundles all required runtime dependencies:
|
|
|
|
```shell
|
|
$ scripts/build_docker_image.sh
|
|
```
|