DeepSeek V4 Flash Switches to 8 Harnesses: Pi Agent Has the Highest Success Rate and Lowest Cost, Claude Code Is the Fastest but Most Expensive

According to monitoring by Beating, AI Agent infrastructure company Composio integrated DeepSeek V4 Flash into 8 different Harnesses and tested the same batch of 30 Agent tasks. Harnesses are execution frameworks outside the model, responsible for context management, tool invocation, and task execution. The tasks involved real-world applications such as Gmail, GitHub, Slack, Calendar, and Notion. The success rates for the 8 Harnesses were as follows: 1. Pi Agent: 20/30 2. Oh My Pi: 17/30 3. Claude Code: 16/30 4. Codex: 16/30 5. Deep Agents: 16/30 6. Hermes Agent: 15/30 7. Prime Agent: 15/24 8. OpenCode: 14/30 Simply switching Harnesses allowed DeepSeek V4 Flash to improve from passing 14 tasks to 20 tasks. The cost and speed differences were also significant. Pi Agent cost approximately $0.028 per successful task, while Claude Code cost $0.195—nearly 7 times higher. Claude Code was the fastest, with a median execution time of 122.7 seconds, while Oh My Pi was the slowest at 272.4 seconds. However, Pi's testing configuration differed from the unified conditions: it used "high" inference intensity instead of "max," and 24 out of the 30 tasks were routed through DeepSeek's official API rather than OpenRouter. Therefore, this performance gap may not be entirely attributable to the Harness. Previously, Composio conducted another round of Harness benchmarking using Kimi K3. At that time, Oh My Pi ranked first with a success rate of 22/25, while the original Pi Agent achieved 18/25. OMP itself is a fork of Pi, enhanced for coding tasks with added capabilities such as sub-Agents. However, after switching to DeepSeek V4 Flash, the original Pi Agent took the lead. [Original Link]