OpenAI misled the entire AI community for several years!

For the past five years, the entire AI industry has been pushed forward by the Scaling Law.

Ultraman firmly believes that the foundation of AGI comes from this curve.

Now, someone has stepped forward to say: this curve was wrong from the very beginning.

This is not hindsight. The person making this statement is researcher Diogo Almeida, who worked on large model optimization at OpenAI back in the day.

Just now, he published a blog post with a chilling title—“Scaling Laws, Honestly.”

The first line kills it: the initial version of the scaling law is wrong because there is a bug.

Link: https://www.completeskeptic.com/p/scaling-laws-honestly

That guy from DeepMind who became famous for diffusion models, Sander Dieleman, immediately boosted it on Twitter, saying this is an interesting story from LLM history:

The original scaling law was wrong due to a bug, likely wasting a massive amount of computing power in the industry on a bunch of "oversized, undertrained" models.

One bug, two years of waste.

When the bug is exposed, we see not only a black hole of computing power but also an intelligence boundary that language itself has reshaped, which is far deeper than we imagined.

The Scaling Law is in fact the LLM version of "Geocentrism"

In 2020, OpenAI concluded: within a fixed computing budget, you should prioritize making the model larger rather than feeding it more data.

In formula terms, the optimal number of parameters is proportional to the 0.73 power of the computing power—parameters are the variable that should be pushed harder.

This statement directly defined the appearance of the GPT-3 generation. Stack parameters. Stack them to death. 175 billion.

It told developers around the world: don’t ask, just stack parameters; as long as you make the model big enough, miracles will happen.

Two years later, DeepMind released Chinchilla, overturning this conclusion: models and data should be scaled up together equally, roughly one parameter per 20 tokens is cost-effective.

They trained a 70 billion parameter Chinchilla, fed it 1.4 trillion tokens—less than half the size of GPT-3, but more than four times the data.

The result, with the same computing budget, completely surpassed the 280 billion parameters of Gopher, which was only fed 300 billion tokens.

In plain language: with the same amount of money, one became a "flabby" strongman, and the other became a lean boxer.

Three years into updates, Peking University alumnus Weng Li delved into the mainstream explanations for the differences in the subsequent research, namely that the difference lies in their method of calculating the total number of parameters.

And this is not the end. Even the "correct" Chinchilla is not clean.

In 2024, Besiroglu and others referred back to the original data points of Chinchilla, discovering that even its fitting had a bug:

The loss scale in the optimizer was set too high, averaging Huber loss per sample instead of summing, leading to premature termination of fitting.

The paper correcting the bug came with another bug.

At this point, the frequently cited "first principle" suddenly seems a bit shaky.

The so-called Scaling Law has never been an ironclad physical law like Newton's three laws; it is merely a curve derived from empirical fitting.

When Diogo Almeida believes the truth is not so, it is not that the methods differ, "but that the initial version of the scaling law itself had a bug."

Did OpenAI fool the global AI community with three tricks?

To create a lie that the global AI community collectively believes, you only need three steps.

First step: imprison the data.

OpenAI's papers fed all models—whether they are still learning to walk (small models) or have already grown into giants (large models)—the exact same "meal." About 130B tokens of data.

As a result, small models were "well-fed" or even "overstuffed," while large models, which genuinely needed massive amounts of data to fill their capacity, were severely malnourished under the same token budget.

The Chinchilla paper later pointed out bluntly: they used a "fixed number of training tokens and learning rate scheduling for all models." (fixed number of training tokens and learning rate schedule).

This is like letting kindergarten kids and doctoral students take the same exam at the same time, then claiming "results are only related to talent."

Second step: covering one’s ears while stealing a bell with LR decay.

They used cosine learning rate decay, making the learning rate smoothly approach zero as training neared its end.

As the training neared the preset endpoint, the learning rate was artificially pressed down to zero little by little, and the model's progress naturally "flattened" out.

When the curve flattened out, it appeared as if the model had maxed out its learning, and further feeding was useless.

The researchers then concluded: "Adding data is useless, the model is already saturated."

This is not the limit of the model; it is the learning rate artificially cutting off the model’s path of growth. It creates a perfect illusion: performance has reached a ceiling, and adding data is useless.

But we now know that those large models hadn't reached their limit at all.

Third step: the arrogance of authority.

The third step, and the most insidious: the paper stated that it was "largely independent of learning rate schedule."

While many, including Diogo Almeida at OpenAI, vaguely sensed something was off, under the fixed token limit, this conclusion was technically correct.

But it did not apply to the ideal world of "infinite data" that the scaling law truly wanted to describe.

They mistook a local truth under limited conditions for a universal cosmic law.

Stacking the three steps together yields a law that is both wrong and extremely difficult to debug.

Even Diogo himself admitted: back in the day, while optimizing at OpenAI, he did not see this bug—after all, that learning rate curve looked too much like it was "carefully set"; who would suspect it.

GPU wasted, serious computing power mismatch

Guided by OpenAI's erroneous formula, the AI industry entered an era of "great efforts yield miracles."

This means that in the past few years, the world's smartest minds and the most scarce computing power have been wasted on ineffective scale expansion.

This is not just about money; it is about humanity collectively running thousands of kilometers down the wrong track due to learning rate settings in the race toward AGI (Artificial General Intelligence).

If the discovery of the bug caused heartache, the subsequent deep reflection sends chills down the spine.

Researcher Adam Zachary Wasserman pointed out a blind spot that everyone overlooked: Even if the formula is corrected, the current Scaling Law is still just "English Scaling Law."

He conducted a counterintuitive experiment: trained models using the same architecture and computing power.

The result found that the French model achieved certain grammatical capabilities far more efficiently than the English model—by 50 to 100 times.

Why? Because English is a "morphologically poor" language.

It relies heavily on distribution, requiring models to guess word meanings from massive data; whereas languages like French and Chinese, which are morphologically rich or structurally tight, carry a lot of clear information within their vocabulary.

This means that all our current computing power distribution schemes are based on the most "data-hungry," least efficient language.

When you think you are exploring the "physical laws of general intelligence," you are actually just measuring "how much computational power is wasted by the English language."

It is like trying to set universal biological nutrition standards by studying a pig's appetite—this is not only biased but also a limitation of cognition.

We could have achieved stronger performance with smaller models and more high-quality data.

We could have saved thousands of running hours of H100 energy and heat.

We could have entered the "efficient AI" era two years earlier.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。