[AI Programming Benchmark Scores Inaccurate Again, OpenAI Finds About 30% of SWE-Bench Pro Questions Problematic] According to monitoring by 动察 Beating, OpenAI has released a programming evaluation audit report, stating that approximately 30% of the questions in SWE-Bench Pro are unsuitable for effective evaluation. SWE-Bench Pro was originally designed to test the real programming capabilities of AI agents and was previously recommended by OpenAI as a replacement for the SWE-bench Verified evaluation set. OpenAI audited 731 publicly available questions in SWE-Bench Pro. Automated processes flagged 200 problematic questions, accounting for 27.4%, while manual review identified 249 issues, accounting for 34.1%. The main problems include incomplete question descriptions, overly strict testing requirements, hidden tests checking for additional criteria, and insufficient test coverage. These issues can distort model scores: failures may not necessarily indicate the model's inability, and passing may not necessarily mean the problem was genuinely solved. OpenAI noted that the pass rate of cutting-edge models on this set of public questions increased from 23.3% to 80.3% over eight months. However, given the noise in the questions themselves, such score improvements should be interpreted with greater caution. OpenAI has withdrawn its previous recommendation to use SWE-Bench Pro and is calling on the community to rebuild a more reliable programming evaluation benchmark. [Original Article Link]