Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul

Summary

When we talk about LlamaIndex, we’re actually referring to an ecosystem consisting of more than 650 Python packages, mostly Integrations and Packs. All these packages share a single GitHub repository, what engineers fondly call a “monorepo”. In this article, we’re going to introduce LlamaDev, our new tool for managing monorepos at scale, and explain the challenges we ran into with existing tooling to get us to this point.The challenge: 650+ dependency treesEach Python package in the monorepo is published on PyPI and comes with its own pyproject.toml file. For those not familiar with the ecosystem, a pyproject.toml file for a modern Python package is a one stop shop defining several aspects of the package lifecycle: its dependencies, its release version number, the Python and operating system versions supported, and how tools like linters and type checkers should behave with that specific package. Most packages have tests, and some have an additional list of development dependencies.Integrations and Packs have some code in common, shipped through a foundational package called llama-index-core that pretty much every package in the ecosystem depends on. Additionally, they can depend on each other, for example llama-index-llms-azure-openai requires llama-index-llms-openai to work. As you can imagine, this has a few implications for the testing strategy, such as:If we touch llama-index-core in a pull request, tests for most of the packages in the monorepo need to be triggered to ensure they keep working after that changeIf we touch something like llama-index-llms-openai, we need to find which are the packages depending on it and test them to ensure they still workDeveloping is easy(ish)The experience of developing in one of these packages varies. Sometimes we work on a single, self-contained Integration with no additional LlamaIndex dependencies other than llama-index-core. That’s the happy path, where we test our code against a certain version of the core package and if...