The AI Industry’s Scaling Obsession Is Headed for a Cliff

A new study from MIT suggesting that the largest and most computationally intensive AI models may soon yield lower returns than smaller models. By mapping the scaling law against continued improvements in model efficiency, the researchers found that performance jumps from giant models may become more difficult while efficiency gains may make models running on more modest hardware increasingly capable over the next decade.

“In the next five to 10 years, things will start to get very compressed,” said Neal Thompson, a computer scientist and MIT professor involved in the research.

Efficiency, like those met with Dipsik’s excellent low-cost model In January, AI already served as a reality check for the industry, which is used to burning through massive amounts of computation.

As it stands, a frontier model from a company like OpenAI is currently much better than a model trained with a fraction of the compute from an academic lab. While the MIT team may not have predictions if, for example, new training methods such as reinforcement learning produce surprising new results, they suggest that big AI firms will have less of an edge in the future.

Hans Gundlach, a research scientist at MIT who led the analysis, became interested in the subject because of the intractable nature of running sophisticated models. Along with Thompson and Jason Lynch, another research scientist at MIT, he mapped the future performance of frontier models compared to models built with more modest computational means. Gundlach says the predicted trend is particularly pronounced for rational models now in vogue, which rely heavily on extra calculations during estimation.

Thompson says the results show the value of honing an algorithm as well as scaling the computation. “If you’re going to spend a lot of money training these models, you should definitely spend some of it trying to develop more efficient algorithms, because that can be very important,” he adds.

The study is particularly interesting given today’s AI infrastructure boom (or should we say “bubble”?) — which shows little sign of slowing.

There is OpenAI and other US technology companies Signed contracts worth hundreds of billions of dollars To build AI infrastructure in the US. “The world needs to compute a lot more,” said Greg Brockman, president of OpenAI. announced this week He announced a partnership between OpenAI and Broadcom for custom AI chips

A growing number of experts are questioning the fairness of this agreement. Roughly 60 percent The cost of building a data center goes toward GPUs, which depreciate quickly. Partnerships are also seen among major players round and opaque.