Uv and Ray: Pain-Free Python Dependencies in Clusters

Summary

When we discovered the uv package manager, we realized that a new era of Python package management had arrived. It tackles the Python developers’ dependency headaches with consistency and speed, which also enables quick development cycles.Here is a (non-exhaustive) collection of things the folks from astral.sh are getting right that we really like:They package the whole environment: You don’t need to worry about setting up your own Python distribution which in itself can be quite cumbersome. Instead, let uv manage the Python environment. If you prefer, uv also supports using a system installed Python.They make uv lightning fast: For starters, it’s written in Rust, just like their very successful Python linter ruff. Designed with speed in mind, uv quickly downloads packages, caches them locally on disk, de-duplicates dependencies between environments and uses hard links to bring them all together.They respect and improve on existing Python conventions: This includes support for pyproject.toml, outstanding platform independent lockfile support, editable packages, environment variables and lots more – more capabilities get added and bugs fixed all the time.However, in a distributed context, managing Python dependencies is still challenging. There are many processes to manage on a cluster, and we need to make sure all their dependencies are consistent with each other. If you change a single dependency, you have to propagate that change to every single other node since each worker process needs the same environment so that all the code gets executed the same way. If there’s a version mismatch somewhere, you could be spending lots of hours debugging what went wrong… which is definitely not a fun way to spend your time. The traditional way to solve this problem is by containerizing everything, but that makes the development iteration much slower - if code and dependencies change, a new version of the container needs to be built and pushed to all nodes. Sometimes the whole ...