Mobilizon-Reshare-condividi-sui-social/doc/dependency-hell.md

Beating dependency hell with GNU Guix

mobilizon-reshare's distribution process relies quite a bit upon GNU Guix. It's involved in our CI pipeline, and it builds the OCI compliant container image available on Docker Hub. It provides us with inspectable, bit-for-bit reproducible and fully bootstrappable images which in turns allows for strong control on what code is actually bundled within the image and it should prevent entire classes of supply-chain attacks starting from the Trusting Trust attack up until the many recent attacks to many FOSS software registries.

To allow for interoperability with the Python ecosystem, we also ship a pyproject.toml that we handle with Poetry. The next paragraph will elaborate on the interactions between Poetry and Guix.

Update the dependency graph of mobilizon-reshare

Beware! - Dependency updates are better delivered to master as a single commit, to avoid confusing the CI.

Python dependencies

We must keep Poetry and Guix version as much aligned as possible, to prevent unpredictable behavior. All the following content assumes this invariant.

Everything starts from pyproject.toml: usually your IDE warns you about outdated dependency, so let's assume you want to bump the version of a Python package. First keep in mind that Poetry's tilde requirements are SemVer compatible but stricter than caret requirements so they should make matching Guix version easier. Then it's time to actually edit pyproject.toml and bump the version of a package.

To update Python dependencies and test your changes the steps are:

$ poetry update
$ guix time-machine -C channels-lock.scm -- build -f guix.scm
$ scripts/build_docker_image.sh

If these steps succeed you can safely commit your changes. If Guix fails you have to examine the output of the command that failed and figure out the problem. 99% of the times it'll be a version mismatch, as Guix's python-build-system has a sanity-check phase that'll try to instantiate the entry point generated by Poetry that, among other things, checks for runtime dependencies versions and errors out if it finds a mismatch between the version actually available in the runtime environment and the version defined in pyproject.toml.

You now have two alternatives:

1. You try to follow the next step about system dependencies. channels-lock.scm locks everything: as long as the Guix commit specified in that file does not change, guix time-machine will look for the exact same package graph. This means that every time we build the image we get the same exact dependencies we ask for, but this semantics is slightly different from Poetry's lock-file which instead tracks the latest version (within the constraints) available on Pypi. Having a more updated Guix version may allow for a more updated mapping of Pypi.
2. You find the package (or packages) responsible for the mismatch and try to manipulate it to follow Poetry's constraints. This requires some basic Scheme understanding but nothing complex. There are many ways a Guix package can be programmatically manipulated as it's just a structured Scheme record, you can start by looking into package variants or also directly at the channel code.

System dependencies

Python's own dependencies are dependencies too! Guix freezes the whole dependency graph of an artifact with channels specifications so to update "system" dependencies you need to follow these steps.

First let's update our Guix version to the latest commit:

$ guix pull
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to d41c82b (162 new commits)...
Building from these channels:
  guix      https://git.savannah.gnu.org/git/guix.git   d41c82b
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%\

[...]

building package cache...
building profile with 3 packages...
$ 

Channels specification define the Guix commit that should be used to fetch the right dependency graph, so what we want to do is replace the commit in channels-lock.scm with the one we just pulled:

$ guix describe
Generation 31   Mar 12 2022 12:35:00    (current)
  guix d41c82b
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: d41c82b481fd0f5c7d45d6e2629fdf9d2085205b
    
$ vim channels-lock.scm

To test our change we can run:

$ guix time-machine -C channels-lock.scm -- build -f guix.scm

But a better test would be to build the Docker image, as that actually bundles all required runtime dependencies:

$ scripts/build_docker_image.sh