Dataframely: A polars-native data frame validation library

Summary

At QuantCo, we are constantly trying to improve the quality of our code bases to ensure that they remain easily maintainable. More recently, this often involved migrating data pipelines from pandas to polars in order to achieve significant performance gains.At the end of 2023, we started undertaking an effort to modernize a massive legacy codebase in our one of our longest-running projects. While doing that, we realized that our existing data frame processing code had an integral flaw: column names, data types, value ranges, and other invariants — none of it was obvious just from reading the code.As a result, the typical approach for understanding a function's behavior involved executing it on client infrastructure — the only place the actual data is available. Then, we would manually step through each pandas transformation to inspect the data before and after every change. Naturally, this is tedious, error-prone, and far from efficient.Once we'd rewritten a chain of transformations in polars, the absence of static type checking or runtime validation on data frame contents meant that bugs were hard to catch. To ensure correctness, we often had to run our entire pipeline end-to-end on large datasets - which required significant time and compute resources.Eventually, we realized that we needed a better way to describe, validate and reason about the content of the data frames in our data pipeline. We wanted to make invariants obvious while reading the code and actually enforce these invariants at runtime to ensure correctness.Data frame validation to the rescueA natural solution to this problem are data frame validation libraries. Already back in 2023, Python libraries existed that allowed defining data frame schemas and verifying that data frames comply with these schemas, i.e. fulfill predefined expectations.In some projects, we had already been using pandera, a widely known open-source library, to validate pandas data frames. Unfortunately, back in 2023, pandera did...