Author: Robonaissance Research Team

Translated by: Deep Tide TechFlow

Deep Tide Introduction: In the increasingly fierce competition of major mainstream large models for scoring, "Model A scores 8 points higher than Model B" is often interpreted in a simplistic and crude manner by the tech community and capital markets as a measure of intelligence. However, the latest empirical research from the Google Cloud AI Research team and Northwestern University completely shatters this myth: without touching any of the underlying weights of any models, merely by restructuring the external workflow framework (Harness), originally underperforming models can easily surpass top models in office and professional tasks.

This discovery reveals a long-ignored brutal reality: the scores and capabilities of large models in real scenarios are not purely determined by the lower-level models themselves but are driven by a comprehensive environment system built by system prompts, memory, skill tools, and collaborative scripts. The so-called scoring rankings are essentially not an intellectual duel between two bare brains but a comparison of performance between two individuals in two differently configured office environments. Understanding this is of great warning value to the current industry ecology that blindly worships large model scores while ignoring application engineering architecture.

An AI agent was assigned a programming task and exhibited an extremely diligent attitude.

It meticulously explained which code file it intended to open, which terminal command it was preparing to execute, and which verification checks it planned to perform.

However, the real fatal trouble was that it merely stayed at the level of "talking about it." The agent simply listed all tool calls in plain text, much like an apprentice standing by with his hands in his pockets and verbally guiding repairs, but not actually running any line of code. Ultimately, this task received a score of only 60 out of a perfect score of 100.

Subsequently, researchers only modified one configuration file in the agent's environment. It was a brief tool call guide that explicitly instructed it not to output any false plain text calling commands. The underlying model remained completely unchanged, its neural network weights untouched, and the testing task was the same, yet the final score soared to a perfect 100.

This vivid case comes from a cutting-edge academic paper jointly published by the Google Cloud AI Research and Northwestern University researchers on October 9, 2026. The test data in the paper are based on the authors' experimental records and have not yet undergone complete independent external reproduction.

This seems like a small yet somewhat humorous incident, but it points directly to a profound core proposition: whenever we read promotional slogans stating "an AI model outperformed another in testing," people often visualize two pure brains competing on stage.

However, in reality, the comparison is between two brains each situated within specific system commands, proprietary tools, and working habits, with the engineering optimization efforts poured behind each system being vastly different. The ultimate evaluation score always belongs to the organic combination of these three, and never solely to the model itself.

Now, this layer of scaffold optimization around the model can already be handled by another AI automation. In these practical tasks, the tuning of this outer environment is sufficient to completely overturn the original model rankings. Therefore, when faced with any claim of "Model A crushing Model B," the truly discerning question should be: In what type of system environment is it operating? To what extent did it receive auxiliary support?

Reversal: How Weak Models Defeat Strong Models

The most shocking conclusion of the paper lies in revealing a dramatic reshuffling of rankings.

In an assessment covering 100 office benchmark tests, including code writing, web searching, and document processing, all tasks were scored on a 100-point scale through hidden test cases, file integrity checks, and evaluation by large model judges. In a standard out-of-the-box system environment, the average test score of the domestic open-source model GLM-5 was 64.2 points; while in the same default environment, Google’s Gemini 3.5 Flash had an average score as high as 72.4 points. Based on the data from this evaluation standard, Gemini is undoubtedly the stronger one, leading by about 8 points.

However, merely by performing one round of automated fine-tuning on the surrounding environmental configuration, the average score of GLM-5 jumped directly to 75.7 points. This was not only 11.5 points higher than its initial score, but even 3.3 points higher than the original out-of-the-box state of Gemini.

In a second ready-to-use test configuration, the same pattern repeated: the optimized GLM-5 scored 75.3 points, surpassing Gemini’s 74.5 points. The authors of the paper succinctly summarized: a relatively weak model equipped with superior scaffold systems can completely defeat stronger models residing in a native, rudimentary environment.

This statement is absolutely true, but the real meaning it conveys is more meticulous and restrained than it sounds. This does not imply that the foundational cognitive capabilities of GLM-5 have surpassed those of Gemini. Because if the same system-level tuning were applied to Gemini, its score would similarly rise to 77.5 points.

What this experimental result truly demonstrates is that in office tasks of this kind, the native capability gap between these two models is far less than the scoring gap that can be created by one round of external system engineering tuning. This magnitude of ranking reversal does not even require any surgical intervention on the weights of any model.

Dissecting Model Scaffolding: Office Components Beyond the Brain

In AI engineering, the system environment surrounding the model is vividly referred to as "scaffolding (Harness)," and its essence is far more straightforward than this professional term might imply.

Imagine what additional provisions a new employee might be offered on their first day at work, aside from their clever brain: a written job description, a list of regulations, handover notes from predecessor employees, a thick manual of standardized operating procedures, an operation guide for office software, ready-to-use scripts for automating tedious mechanical tasks, and a team of colleagues for collaborative division of work.

The AI scaffolding is essentially this digital twin of the work package. The paper systematically breaks this down into eight core components: Standing Instructions, Guardrails, Memory, Skills, Tool Guidance, Scripts, Helper Agents, and a lightweight plugin that can dynamically intercept and correct agent behaviors. Each of these is an independent file, and none of them change the underlying model weights. What they change is merely what the model is told, what it remembers, and what it is permitted or reminded to do.

This workplace analogy is equally applicable on another level: placing the same talented individual in a well-organized efficient office versus a chaotic management quagmire often yields completely different assessments of their performance. No one would perceive that test as a pure measure of that employee's individual capabilities; however, currently, when the industry interprets "Model A scored 72 points," it almost invariably falls into this one-dimensional cognitive pitfall.

Adaptive Evolution Without Standard Answers: Correcting Behavior Instead of Grading Right or Wrong

The most intuitive way to optimize a set of scaffolding is, of course, to constantly try modifying configurations and keep those that boost scores. However, this approach assumes there is first a ready-made scoring board, and in the real world, when faced with brand new open-ended tasks, agents have no preset scoring standards at all, for no one can provide a unique standard answer in advance. The innovative method proposed in the paper cleverly bypasses this barrier by comparing the "behavior patterns" of the agent rather than merely judging the results as "absolutely right or wrong."

The single-step optimization mechanism works roughly as follows: the agent first runs a task comprehensively in the existing scaffolding environment, and that run record serves as the reference sample (Reference Run). Subsequently, a model responsible for editing and tuning reviews the entire execution trajectory and proposes two new optimized scaffold plans: the first plan addresses general behavioral weaknesses of the agent, such as task planning, tool calling, and context tracking capabilities; the second plan supplements domain knowledge and rigid business rules specific to that particular professional vertical.

Next, these two modified scaffolds will run the same task again, and their execution processes will be meticulously compared to the initial reference sample: did this modification accurately correct the expected defective behaviors, did it have no effect, or did it inadvertently disrupt other normal processes? In the final step, the system selects and retains only those improvements that have been validated in real combat, consolidating them into a final scaffold that will produce the final deliverable. After the task is completed, the system archives the invaluable experience gained into a case library for future similar tasks to reference as needed.

The official description of this method is particularly honest about the role of the baseline run: it "reveals the behavioral characteristics of the scaffolding, rather than whether its final deliverables are absolutely correct." This is the core honesty of the method. It cannot a priori distinguish whether answers are right or wrong; rather, it captures weaknesses in the execution behaviors themselves, and what is successfully corrected are precisely those visible behavioral flaws.

Let’s examine a few typical failure cases that this method successfully remedied: in a complex professional office task, the agent needed to synthesize a report memo from multiple scattered data source files, including schedules, activity venues, personnel work hours, and financial budgets. Its initial run score was only 43.1 points, and execution logs showed that this agent never cross-verified the numbers between different files and also never reopened to check the generated documents after writing was completed. After automation tuning of the scaffolding, it was forced to list each input basis item by item, track the original source of each number, and run a script to reopen the final document for review, resulting in an immediate rise to a score of 77.8 points.

In another legal practice task, the agent needed to review and annotate an arbitration order. Initially, it saved all modified content in the revision tracking layer of a Word document, causing any reader viewing the file in plain text format to only see the final text after revisions, with annotation traces completely invisible. A newly injected skill package mandated that it directly write addition and deletion markers within the main body of the text, resulting in an instant score jump from 44.4 points to 94.5 points. In a third case, an assistant agent responsible for data collection fell into an endless retrieval loop, leaving it empty-handed as the deadline approached with no deliverable files generated; a protective plugin that issued warnings upon detecting a specific number of pure tool calls completely solved this procrastination issue.

It is evident that none of these defects stemmed from a lack of knowledge reserves in the models. These agents possessed abundant legal texts, arithmetic logic, and coding norms; what they genuinely lacked were good professional habits: carefully reviewing your work output, clearly documenting findings, ensuring that readers can see your modifications, and decisively pressing the execute button when necessary. Professional habits are clearly observable in execution trajectories, as opposed to abstract answers' correctness, which exactly explains why, in two of the three benchmark tests, models that started with relatively weaker foundations achieved greater improvements because they often had to shed more bad habits.

The Cost Behind the Reversal: A Stronger Editor and Additional Computational Expense

It must be noted that the optimized GLM-5 did not win this competition solely by its individual efforts. Its underlying scaffolding was personally rewritten by Claude Opus 4.8, which is significantly superior to any competitor among the two contestants; moreover, before producing the final deliverable, each task was run multiple times: one baseline run, one for each of the two candidate solutions, and one for the merged final version. In contrast, the control group Gemini merely ran a single pass under its standard default scaffolding.

Therefore, the essence of this reversal is that GLM-5 was layered with a more powerful editing mentor and multiple additional runtime expenses, to face Gemini, which had no external assistance. The resources and computational costs consumed by both sides were entirely unequal, and the experimental results themselves did not conceal this fact.

However, two strong empirical pieces of evidence obstruct the tendency to simply attribute this achievement to "all thanks to a stronger model doing the work for it." Firstly, the researchers specifically designed a set of ablation comparison experiments that allowed models to autonomously edit and optimize scaffolding, covering five different tiers of models, from GLM-5 to Claude Opus 5. The test results showed that each model achieved comprehensive improvements in self-scores under this mechanism. This implies that the core logic of this method does not absolutely depend on external stronger mentor models, even though the main experimental data charts chose the strongest model for the editing end in pursuit of the best outcome.

Secondly, according to the authors’ meticulous computational accounting (combining all tokens consumed in the tuning edit process with those actually consumed in executing tasks), within the computational budget testing range they established, this method still outperformed all other similar solutions in performance score output per unit computational expense. This is a rigorous empirical within a specific budget range, rather than proving that this is the best paradigm for exhaustively investing all available computational resources.

It is also crucial to be cautious that modifying scaffolding does not inherently guarantee positive gains. In two other published agent enhancement schemes, scores have occasionally fallen significantly below the unmodified original scaffold scores, with the maximum drop in programming tasks reaching up to 6.6 points; one of the other schemes saw scores drop below the initial baseline in as many as 7 out of 12 sets of benchmark and environmental combinations. Even when using the editor from this paper itself, blindly rewriting scaffolding without closed-loop verification through baseline runs and the case library may result in programming scores plummeting from 73.4 points to 63.2 points. This layer of system structure surrounding large models has a very high leverage effect, capable of causing drastic fluctuations in final performance; and this experiment’s ability to push for stable upward movement is entirely attributable to strict compliance with a hard-and-fast rule: every modification was rigorously validated within the real operational behaviors of the agents.

The Boundaries and Insights of Empirical Conclusions: Scoring is Never a Bare-Brain Duel

Based on the firm boundaries supported by existing data, we can arrive at a clear judgment: in office, professional consulting, and legal tasks, the volume of the scaffold layer surrounding large models is substantial enough to completely reshape the relative winning and losing rankings among models; at the same time, its adjustment costs are low enough to allow for dynamic customization for each specific task. This is precisely everything that this paper has solidly proven.

However, it has not proven that these improvements represent an absolute intellectual leap in a deeper philosophical sense. The evaluations of these tasks largely depend on the scoring by large model judges, and the criteria for judgment in legal tests are also quantified scores rather than perfect achievement of the entire task. The author team itself acknowledges that purely comparing behavioral patterns in the absence of a definitive factual baseline cannot guarantee mathematical correctness nor ensure that any particular modification can seamlessly generalize to entirely different brand-new scenarios. Moreover, the scale of the test sets is relatively limited (with each field containing only 25 to 100 tasks), and the occasional fluctuations of a few tasks can significantly pull the overall average; finally, the data here has not manifested a miracle of self-evolution in a snowballing manner, and the research team did not conduct prolonged task flow stress tests, nor did they claim to have achieved unlimited self-intelligent leaps.

Nonetheless, despite these objective limitations, the direction of technological evolution it indicates is incredibly clear. If a model’s system environment can be autonomously optimized and restructured by another AI within just a few task executions, then the so-called “bare-metal models under the default scaffolding” no longer constitute a natural benchmark unit for measuring AI capability. They merely turn into one of many possible configurations, and often not the type that anyone would genuinely adopt in real production practices.

We cannot expect this layer of engineering scaffolding to vanish simply by waiting for a more perfect foundational large model to be born. Many of the faults repaired in this experiment are tightly interwoven with specific real work environments, such as those underlying revision marks that readers cannot see or research tasks that remain unwritten as deadlines loom, and every entirely new workplace environment brings its unique friction. Even the strongest model evaluated in the paper can only gain a few points after its own system environment tuning. A stronger model may require fewer reminders and corrections, but it ultimately needs to clock in at a specific office, where its operating manual must be written in advance by some individual or some intelligent agents.

The agent that initially could only verbally repeat tool commands on screen has an internal cognitive capacity that is by no means inferior to the later agent that can actually execute the commands. They are entirely the same brain, merely lacking a key reminder in their work directives. The next time you see a ranking chart on social media declaring one large model has completely outperformed another, please be sure to recognize that this list records merely the performances of two brains in two differently configured offices after undergoing a certain number of trial and error processes. Until thoroughly understanding who behind the scenes has decorated these offices for them and how many retake opportunities they each received, the real intelligence information you can glean from that chart is far narrower than it appears.