Why LLMs Can't Build Software

Summary

One of the things I have spent a lot of time doing is interviewing software engineers. This is obviously a hard task, and I don’t claim to have a magic solution; but it’s given me some time to reflect on what effective software engineers actually do. When you watch someone who knows what they are doing, you'll see them looping over the following steps: Build a mental model of the requirements Write code that (hopefully?!) does that Build a mental model of what the code actually does Identify the differences, and update the code (or the requirements). There are lots of different ways to do these things, but the distinguishing factor of effective engineers is their ability to build and maintain clear mental models. To be fair, LLMs are quite good at writing code. They're also reasonably good at updating code when you identify the problem to fix. They can also do all the things that real software engineers do: read the code, write and run tests, add logging, and (presumably) use a debugger. But what they cannot do is maintain clear mental models. LLMs get endlessly confused: they assume the code they wrote actually works; when test fail, they are left guessing as to whether to fix the code or the tests; and when it gets frustrating, they just delete the whole lot and start over. This is exactly the opposite of what I am looking for. Software engineers test their work as they go. When tests fail, they can check in with their mental model to decide whether to fix the code or the tests, or just to gather more data before making a decision. When they get frustrated, they can reach for help by talking things through. And although sometimes they do delete it all and start over, they do so with a clearer understanding of the problem. Will this change as models become more capable? Perhaps?? But I think it's going to require a change in how models are built and optimized. Software engineering requires models that can do more than just generate code. When a person runs into a p...