Large Language Models Are Improving Exponentially

Summary

Benchmarking large language models presents some unusual challenges. For one, the main purpose of many LLMs is to provide compelling text that’s indistinguishable from human writing. And success in that task may not correlate with metrics traditionally used to judge processor performance, such as instruction execution rate.RELATED: LLM Benchmarking Shows Capabilities Doubling Every 7 MonthsBut there are solid reasons to persevere in attempting to gauge the performance of LLMs. Otherwise, it’s impossible to know quantitatively how much better LLMs are becoming over time—and to estimate when they might be capable of completing substantial and useful projects by themselves. Large Language Models are more challenged by tasks that have a high “messiness” score.Model Evaluation & Threat ResearchThat was a key motivation behind work at Model Evaluation & Threat Research (METR). The organization, based in Berkeley, Calif., “researches, develops, and runs evaluations of frontier AI systems’ ability to complete complex tasks without human input.” In March, the group released a paper called Measuring AI Ability to Complete Long Tasks, which reached a startling conclusion: According to a metric it devised, the capabilities of key LLMs are doubling every seven months. This realization leads to a second conclusion, equally stunning: By 2030, the most advanced LLMs should be able to complete, with 50 percent reliability, a software-based task that takes humans a full month of 40-hour workweeks. And the LLMs would likely be able to do many of these tasks much more quickly than humans, taking only days, or even just hours.An LLM Might Write a Decent Novel by 2030Such tasks might include starting up a company, writing a novel, or greatly improving an existing LLM. The availability of LLMs with that kind of capability “would come with enormous stakes, both in terms of potential benefits and potential risks,” AI researcher Zach Stein-Perlman wrote in a blog post.At the heart of the MET...