The comparison of Sakana Fugu and Fable 5 running points has been questioned, and the difference in test scaffolding can cause a 10-20 point deviation According to Beating monitoring, Fugu Ultra, a multi-agent collaboration system developed by Japanese AI startup Sakana AI, claims to have defeated the Anthropic flagship model Fable 5 in multiple benchmark tests such as scientific reasoning and programming. However, the scoring conclusion has been widely questioned by the community. Critics point out that comparing self-test data in non-uniform examination rooms is not objective. The test score is highly dependent on the running scaffold (Scaffold/Harness), and the deviation in score caused by different scaffolds can reach 10 to 20 points. This makes the so-called "surpassing" largely a product of system engineering optimization, rather than a cross generational surpassing of underlying model capabilities. Independent evaluation data shows that the scaffolding of intelligent agents built around large models has a significant impact on the final score. Under the same Claude Opus 4.5 model, replacing only three different open-source scaffolds resulted in a 50.2% to 55.4% fluctuation in repair rates in the SWE bench Pro benchmark test. The analysis by Scale AI, a third-party testing agency, further confirms that operational strategies such as prompt word templates, maximum number of attempts, context retention management, and tool call integration are sufficient to cause a 10 to 20 point deviation in the weight of the same set of models. Due to the fact that the data released by Sakana AI and Anthropic are both based on vendor scaffolds specifically tuned for their own systems, and have not been uniformly tested in standardized independent third-party environments such as Scale SEAL, the data cannot truly reflect the strength of the underlying capabilities of the two models. [Original link]