If you just want the solution, jump to the final script

I recently started a new python project and decided that I had enough of requirements.txt and standard venvs. As such I set up the project with poetry, a Python packaging and dependency management system that relies on pyproject.toml for dependency configuration. (If you’re unfamiliar with poetry, their introduction page explains it quite well).

The main issue with using poetry in a team, is that if one person adds a dependancy then the entire team needs to remember to run poetry lock --no-update || poetry install which is always a good time when you’re working with team mates who were quite frankly against using poetry in the first place. Poetry, being a python tool, gives you absolutely no help whatsoever when your builds / runs start failing and your imports / versions are incorrect.

So the goal here is to find a way that will update your venv (and resulting poetry lock) whenever a teammate pushes a new dependency, or changes the minimal pinned version of a dependency.

pre-commit to the rescue⌗

Pre-commit is an absolutely stellar tool, allowing one to configure all kinds of multi language git commit hooks for a project using yaml, which it then installs into virtual environments and symlinks in to your project’s git hooks folder.

Now we can’t just register a standard pre-commit hook, we’re trying to update our dependencies when we pull someone else’s changes, not when we push our own. Thankfully, pre-commit allows us to register any kind of defined git hook, including the ones we’ll need for our fix.

pre-commit allows us to create our own “local” git hooks without pulling any predefined ones from GitHub/GitLab, so that’s what we’ll get started with.

- repo: local
  hooks: 
    - id: poetry-lock-and-update
      name: "Update poetry lock and install"
      stages:
        - ????

Now when do we run our newly created (completely empty hook)?

A quick insight to pulling, switching and rebasing⌗

Git allows registering hooks for all kinds of actions (if you wanna read all of them, go take a look at the Git Hooks doc page), but oddly enough, pulling is not one of them. Now this is largely due to how many different ways one can pull a branch, pull can use merge, or it can use rebase when melding in upstream changes into your working branch, so our solution will actually have to use a multitude of different hooks.

There are 3 specific scenarios where we would want to update our local lock file:

• post-checkout: We’ve switched to a new branch with different dependencies, we want to make sure that our poetry.lock is consistent with the pyproject.toml (and as such the resulting venv has the correct dependencies).
• post-merge: We’ve pulled a branch, it’s been merged into our current working branch, and the pyproject.toml has changed
• post-rewrite: This is the strange one. Post-rewrite is triggered when the history of a branch is rewritten, which doesn’t seem like something we would encounter very often. However, git pull –rebase (which is a lot of people’s default option) will rewrite the branch history to include the pulled changes instead of merging, and as such the post-merge hook would never get triggered.

For good measure we’ll also add post-commit incase any existing hooks or actions would cause our poetry.lock to become inconsistent with the pyproject.toml file. Pre-commit also allows for a manual hook which just allows you to run it whenever one might feel like it, so we’ll add that as well, just for ease of testing.

Normally pre-commit allows for choosing what hooks need to be ran when a certain file / filetype is changed, however, with the post-* hooks we need to tell it to run at any file change as it will not detect those files as changed.

- repo: local
  hooks: 
    - id: poetry-lock-and-update
      name: "Update poetry lock and install"
+     always_run: True
+     stages:
+       - "post-commit"
+       - "post-checkout"
+       - "post-merge"
+       - "post-rewrite"
+       - "manual"

What do we run?⌗

As we said before, the commands are pretty simple to update your lock file, it’s just a quick

poetry lock --no-update
poetry install

This has one rather large caveat. poetry lock --no-update can take a while to run as it needs to resolve all the dependencies. Not normally much of an issue, but if you’re someone that’s hopping between branches all day you’re going to be sitting there twiddling your thumbs quite a lot. There’s got to be a smarter way of doing things!

Poetry gives us the command poetry lock --check which will return a status of 0 if the current poetry.lock is consistent with the pyproject.toml file. Adding the --quiet flag surpasses standard output and lets us rely purely on the status code to know if it’s worth updating our lock file. So, with a simple bash script we can get a quick short-circuiting way to update our lock file only when the two stop being consistent. Lets chuck this into a file called lock_and_update.sh and make it executable with chmod +x ./lock_and_update.sh

#! /usr/bin/env bash
if ! poetry lock --check --quiet; then 
    poetry lock --no-update && poetry install
fi

Lets chuck that into a file called lock_and_update.sh and make it executable with chmod +x ./lock_and_update.sh

Now we need to invoke it in our .pre-commit-config.yaml, specifying that it’s an executable script, and that it doesn’t take any file names as arguments.

- repo: local
  hooks: 
    - id: poetry-lock-and-update
      name: "Update poetry lock and install"
      always_run: True
      stages:
        - "post-commit"
        - "post-checkout"
        - "post-merge"
        - "post-rewrite"
        - "manual"
+     language: script
+     entry: lock_and_update.sh
+     pass_filenames: false

Now the only thing left to do is to tell pre-commit that you want it to install all the possible types of githook, as normally it will only install pre-commit hooks.

default_install_hook_types:
  ["pre-commit", "post-checkout", "post-merge", "post-rewrite"]

- repo: local
  hooks: 
    - id: poetry-lock-and-update
      name: "Update poetry lock and install"
      pass_filenames: false
      language: script
      stages:
        - "post-commit"
        - "post-checkout"
        - "post-merge"
        - "post-rewrite"
        - "manual"
      entry: lock_and_update.sh
      always_run: True

Now with 21 lines of code you have all of your teams repo’s dependencies automatically updating to the supported versions when someone adds a new dependency.

All that’s left now is to run pre-commit install and you’re good to go! One caveat is that your entire team needs to run pre-commit install, but it’s a lot easier than having them remember to run poetry lock --no-update every time.

Final result⌗

Bash script lock_and_update.sh

#! /usr/bin/env bash
if ! poetry lock --check --quiet; then 
    poetry lock --no-update && poetry install
fi

.pre-commit-config.yaml:

default_install_hook_types:
  ["pre-commit", "post-checkout", "post-merge", "post-rewrite"]

- repo: local
  hooks: 
    - id: poetry-lock-and-update
      name: "Update poetry lock and install"
      pass_filenames: false
      language: script
      stages:
        - "post-commit"
        - "post-checkout"
        - "post-merge"
        - "post-rewrite"
        - "manual"
      entry: lock_and_update.sh
      always_run: True

Run pre-commit install and you’re good to go!