Shrinking Elephants: Managing large projects with Gradle

Summary

Photo by Lili Koslowski on Unsplash. So you want Intellij IDEA to sync 5 million lines of Kotlin code,1 spread across more than 2000 Gradle projects. No, you don't. In one of our largest repos, a Kotlin backend project, syncing it all takes an average of about 8.4 minutes, and that's being generous. Achieving that required a fully up-to-date build, with all dependencies already downloaded—meaning this was 8.4 minutes for what is essentially a no-op. This no-op sync used the full 24 GiB of heap provided to the Gradle daemon, and an additional 12 GiB heap provided to the IDE process, or 36 GiB in total. If this is the best you can offer your developers, do not expect them to be very productive, and do not expect to ship software quickly. A more realistic scenario is triggering the first sync of the day after a git pull that updated dependencies. In that "cold sync" scenario, the same build described above took 24.7 minutes, more than 7 minutes of which was spent downloading over 26000 files (jars plus metadata) totalling nearly 4 GiB over the wire. This build maxed out the heap very quickly, and it wouldn't be hard to imagine a scenario where the build thrashed gc for an hour before finally OOMing. While you may want the full project loaded into memory, you almost certainly can't bear the consequences: extremely long sync, sluggish editing, the spinning beachball of death… Fortunately, our developers never have to do this (sometimes a build engineer might… for analysis purposes). The actual experience we target for our developers is benchmarked at 15 seconds, a 97% improvement over the baseline outlined above; and only using 3.5 GiB heap, a 75% improvement. How did we do it? What is IDE sync? Gradle Sync is how the IDE loads important information about your build (known as the project model) to enable smart editing features like autocomplete, code navigation, viewing sources for dependencies, etc. To make this work, IntelliJ injects a library into the Gradle daemon co...