AI code review: Should the author be the reviewer?

Summary

Alternate title: I’m using AI to write code. Is it silly to use AI to review it? I'm Daksh, a co-founder of Greptile. Our product uses AI to review pull requests to surface bugs and anti-patterns that humans might miss. Here is an example of what that looks like. Recently, I was curious to see if there exists a power law in the number of PRs opened by individual Greptile users. In other words - were some users opening orders of magnitude more PRs than others? A quick SQL query later, I discovered that there is a power law to this. I also noticed something else very interesting: At the far left of the long list of GitHub usernames was “devin-ai-integration[bot]”. An AI bot was writing more pull requests than any individual human. [1] Seeing as Devin uses the same LLMs under-the-hood as Greptile, it does raise an interesting question - should the author be the reviewer? [1] Granted that this is somewhat of a technicality. Devin’s contributions across many orgs are being counted in aggregate here. It would be more accurate to treat “Devin @ Company A” and “Devin @ Company B” as separate entries in this chart. Should the Author Be The Reviewer? Most software companies wouldn’t want the PR reviewer to be the same person as the PR author. A large part of why PR reviews happen is to ensure every new piece of code is getting a fresh set of eyes. It seems silly to have Claude Sonnet generate a bunch of code, and then expect Claude Sonnet to find bugs in it. There are a few counterpoints worth discussing: Statelessness If you’ve used LLM APIs, you’ll notice that they are stateless. Every inference call is a clean slate request for intelligence. As a result - asking an LLM to review its own code is looking at it with a fresh set of eyes. Scaffolding Scaffolding refers generally to the specific workflows that a tool uses to wrap the LLM call to allow it to do the task at hand. For an AI code reviewer it might be the set of steps it takes to review a diff, checking for bugs, for...