Scala 3 slowed us down?

Summary

Is this clickbait? Not really. Is this the fault of the language or the compiler? Definitely not. Rather, it was part of a rushed migration. Sharing the lessons learned in the process. I was refreshing one of our services. Part of this process was to migrate codebase from Scala 2.13 to Scala 3. I’ve done this a few times before and overall had a positive experience. Well, at least until we talk about projects with macro wizardry. The service in question had no macros at all, but it was at the heart of data ingestion, so performance was not an afterthought. I did it as usual - updating dependencies, compiler options and some type/syntax changes. Then after resolving few tricky implicit resolutions and config derivations, project compiled on Scala 3.7.3 🎉 All tests passed, end-to-end flow locally works perfectly fine, so I decided to roll out the changes in a testing environment. Similarly, no issues at all. No concerning logs, all metrics ranging from infrastructure, through JVM up to application level look healthy. With that in mind, I began a staged rollout. Again, all seem good. I kept observing the service but it looked like my job is done. Well, as you probably can guess, it wasn’t. The mysterious slowdown After 5-6 hours, Kafka lag started increasing on a few environments. Of course, this wasn’t something new. Most often it is caused by a spike of data. We have pretty advanced machinery to deal with that. Usually the lag resolves by itself without any manual action. However, this time something was off. Upstream load turned out to be relatively modest, yet we needed much more instances of the service - meaning the processing rate per instance dropped. I was confused to say the least. Why would it decrease the processing rate just on these environments? Anyway, we decided to rollback the changes - this brought the rate back. Digging deeper I came back to testing. In particular, load testing. However similarly as on production environments I did not notice regres...