How to Automate Software Engineering

Summary

How to fully automate software engineering Ege Erdil, Matthew Barnett, Tamay Besiroglu May 30, 2025 With every passing month, AI models get better at most tasks that a software engineer does in their job. Yet for all these gains, today’s models only assist human engineers, falling far short of automating their job completely. What will it take to build AIs that can fully replace software engineers, and why aren’t we there yet? Current AIs present something of a paradox. Their performance on narrow coding tasks already exceeds that of most human software engineers. However, any engineer who has worked with them quickly notices the need to keep AI agents such as Claude Code on a very short leash. Despite good benchmark scores and impressive demos, there are clearly core capabilities that human engineers have that our current systems are missing. We’ve previously highlighted some of these shortcomings: lack of reliability, poor long context performance, and overly narrow agentic capabilities, among others. But why are these capabilities missing in AI systems to begin with? We train them on more compute and data than humans have access to in their entire lives, and we can run tens of millions of parallel copies of them, and yet it’s still not enough. On some level, the answer has to be that our learning algorithms have been and remain much less efficient than the human brain. Deep learning skeptics often point to this and say that it’s a sign the entire paradigm is doomed. We draw a different conclusion. The bitter lesson of the past decades of AI research is that handcrafted algorithms perform poorly, and the best algorithms are the ones that are discovered by applying massive amounts of compute for search and learning. This is the principle that drove the pretraining revolution, where scaling up training on massive text datasets allowed models to spontaneously develop powerful meta-learning abilities. For the past decade of scaling, we’ve been spoiled by the enormous ...