Cursor breaks the myth of model ranking: Opus 60% successful solution relies on copying web pages and digging Git history According to the observation of Beating, the evaluation research released by Cursor shows that when programming agents can access the code base history or the Internet, they often pass the evaluation by directly retrieving the answers, which is called Reward Hacking. To quantify the actual proportion of retrieval cheating, Cursor deployed an audit agent to analyze the 731 running trajectories of Opus 4.8 Max in the SWE bench Pro benchmark test. In successfully repaired cases, 63% of successful solutions come from retrieval rather than autonomous deduction. Among all audited trajectories, 57% found merged PR or fix source files on public web pages and copied them almost verbatim, while another 9% mined future submissions and extracted patches in the packaged. git history. In a strict sandbox environment where the. git directory is cleared, reset to single commit, and network access is restricted, the scores of mainstream models have significantly decreased. The pass rate of Opus 4.8 Max has dropped from 87.1% to 73.0%, a decrease of 14.1 percentage points. The score of Composer 2.5, a self-developed model by Cursor, dropped sharply from 74.7% to 54.0%, a decrease of 20.7 percentage points. The comparison shows that the older Opus 4.6 scores almost unchanged in both the old and new sandboxes, while the more capable new model has a more pronounced tendency to reward hackers for testing environment vulnerabilities. Cursor suggests that when evaluating programming agents, we should not only focus on dataset construction, but also isolate the runtime environment to prevent the model from retrieving external ready-made answers through vulnerabilities. At the same time, the development team should audit the running trajectory of the model during testing to ensure that the score reflects real programming ability rather than search and retrieval skills. [Original link]