Why frozen test fixtures are a problem on large projects and how to avoid them

Summary

Tests grow to thousands All make their claim on fixtures Frozen by demandsAn ancient Japanese Haiku about a common problem with software test fixturesAct 1: The problem, frozen fixturesFixtures have a lot going for them: super fast, clearly structured, reusable across tests …That last one is also the source of a common problem in large test suites. Every time you change fixtures you risk falsely breaking some tests. Meaning: the test fails even though the feature it tests still works. This is because every test makes assumptions about the fixtures. This is necessarily part of the test setup, even if it is not explicit in the test code. If the code breaks those assumptions the test itself will no longer work. The more tests there are, the more likely you are to falsely break some of them when you change fixtures.This is why sufficiently complex data model fixtures tend to become frozen after a certain number of tests. If you aren’t careful, when you get to 1000s of tests, making any change to fixtures can break 10s or even 100s of unrelated tests. It becomes really hard to fix them so you try to avoid directly modifying the fixtures at all. You start to work around it (more on that below) and they stop changing. Hence, frozen fixtures.Thankfully, there are ways to write tests to minimise this effect but it requires discipline.Act 2: The bad solutionsFirst, let me go over 2 approaches I’ve seen on projects and why I think they’re bad:If current fixtures can’t be reused, create new ones. This is especially prominent in multi-tenant applications: create a brand new tenant in fixtures just for the new tests you’re adding. This is a road of ever increasing fixture size. It becomes really hard to understand which fixtures are for which tests and the testing database starts to become larger and larger. Reviewing existing fixtures for reuse becomes harder. It becomes easier to just add new fixtures for the next test which makes the problem worse.Use code inside the test to m...