[Crossing the Red Line Means Zero: Zapier Launches AI Automation New Benchmark, Top Model Scores Halved] According to monitoring by Beating, workflow automation platform Zapier has partnered with evaluation agency Artificial Analysis to launch an independent SaaS workflow automation benchmark, AutomationBench-AA. The benchmark is based on real, anonymized data from the Zapier platform and is designed with a private test set that includes 657 multi-step tasks and 40 simulated SaaS software environments (including Gmail, Slack, Salesforce, Jira, etc.) for objective third-party evaluation. The rigor of the test lies in the introduction of compliance guardrails (Guardrails). Large models must complete operations without violating any preset enterprise business rules. If any compliance violations are triggered during execution, the final task score will be directly reset to zero. In the first round of evaluations, large models generally saw their scores plummet due to penalties from compliance guardrails, with no final scores exceeding 50%. Among them, Claude Fable 5, which introduced a "high-difficulty task fallback to Opus 4.8 (approximately 18%)" safety downgrade mechanism, actually completed 73% of the task steps but achieved a comprehensive score of only 48.6%. Gemini 3.5 Flash and GPT-5.5 also completed nearly 70% of the tasks (68% and 67%, respectively), but their final scores dropped to 42.6% and 42.1%. The open-source leader GLM-5.2 scored only 27.8%. The test indicates that how models maintain compliance guardrails during automated operations has become a critical challenge for real-world implementation. [Original Link]