The BinEval framework automatically scores AI using yes/no questions, solving the pain points of false full marks and opacity in the referee model According to the dynamic observation of Beating, the research team of Capital One proposed the BinEval evaluation framework, which automatically disassembled the complex scoring criteria into specific "yes or no" Single choice question to solve the problem of black box scoring and false high scores. The framework allows the evaluation model to answer each yes or no question one by one, and finally calculates the score based on the proportion of correctly answered questions. In the testing of three mainstream datasets, the BinEval scoring quality using large models such as Claude Sonnet 4 matched or surpassed mainstream evaluation tools such as UniEval, and was particularly adept at identifying answers that appeared smooth on the surface but were actually incorrect. Taking a summary evaluation involving aircraft interception as an example, although the summary reads smoothly and the entity and aircraft model are correct, the summary reverses the statements of the Pentagon and Russia, and even fabricates the website. The old AI judges gave a perfect score of 5.0 because they only looked at the surface. And BinEval accurately identified four factual errors based on seven true and false questions, giving a score of 1.57, which is very close to the 2.0 score given by humans. The wrong answer book for yes/no questions can be used to optimize the evaluation criteria of the referee model itself, as well as to automatically modify the writing prompt words. Experiments have shown that feedback optimization can increase the compliance rate of format and sentence structure by 17 percentage points in instruction adherence testing. However, optimization tools are still powerless for hard skills such as word limits that require mathematical calculations, and excessive disassembly requirements can make evaluation standards too strict. [Original link]