What happens when you drop the world's most capable AI models into a test that spans everything from consumer apps to enterprise workflows, across 354 distinct domains, and ask them to actually complete real multi-step tasks rather than answer multiple-choice questions? The answer is sobering: even the best models fail more than 40% of the time. A new benchmark called OmniaBench, released on arXiv by a research team led by Chengyu Shen and Wentao Zhang, puts frontier AI agents through 1,431 tasks spanning consumer, business, and enterprise scenarios. The results reveal a wide gap between what these models can do in controlled chatbot settings and what they can reliably execute as autonomous agents.
Claude-Sonnet-5 leads the pack with an Overall Pass@1 score of 58.54%, followed by GPT-5.6 Sol at 57.14%. GPT-5.6 Turbo scores 52.65%, and GPT-4.5 achieves 49.37%. No model crosses the 60% threshold. For founders who are building agentic products on top of these models, these numbers should be required reading before the next architecture review. The benchmark suggests that your agentic workflow is likely failing 40% of the time on the model layer alone, before accounting for any bugs in your own toolchain, prompt engineering, or error handling.
What OmniaBench Actually Tests
Most existing AI benchmarks fall into one of two categories: narrow academic tests that measure factual recall (like MMLU, GSM8K, or HumanEval) or domain-specific agent benchmarks that test a single tool ecosystem (like SWE-Bench for software engineering or WebArena for web browsing). OmniaBench takes a fundamentally different approach. It starts by building a hierarchical taxonomy of application domains derived from app store data, product documentation, industry resources, web retrieval, and human expert refinement. The resulting taxonomy covers three sectors: ToC (consumer), ToB (business), and ToE (enterprise), with 90 level-1 and 354 level-2 domains.
From this taxonomy, the researchers construct executable environments and generate tasks through four complementary routes: DAG (directed acyclic graph task construction), DAG-S (DAG with state tracking), Solver (a task synthesis method), and Program (programmatic task generation). Each task involves interacting with a tool-based environment, making multi-step decisions, and maintaining context across turns. The results are measured on Pass@1, meaning the agent gets exactly one attempt per task. This is the right metric for production use cases where a failure means a dropped customer request, not an opportunity to retry.
The benchmark also introduces a ten-dimensional capability taxonomy for fine-grained analysis. The dimensions include instruction following, tool use, planning, constraint maintenance, adaptive correction, state tracking, grounding, multi-turn coherence, error recovery, and efficiency. This level of granularity lets builders identify exactly where their chosen model falls short, rather than relying on a single aggregate score.
The Capability Gaps That Matter for Builders
When you break down the results by capability dimension, the picture becomes more useful for decision-making. The strongest capability across all models is instruction following, where frontier models score in the mid-to-high 70% range. The weakest capabilities are planning (where top models barely cross 50%), constraint maintenance (consistently below 45%), and adaptive correction (the ability to detect and recover from mistakes, which even Claude-Sonnet-5 handles correctly only about 40% of the time).
These specific gaps map directly to production failures that founders encounter every day. A customer support agent that follows instructions correctly but cannot maintain constraints will accept orders outside business hours. A code generation agent that plans poorly will produce solutions that work for the immediate prompt but break under edge cases. An agent that cannot adaptively correct its mistakes will compound errors rather than self-correcting, turning a small input formatting issue into a full workflow failure that requires human intervention.
The researchers also identify eight compositional atomic difficulty factors that predict task failure. These include task length, branching factor, dependency depth, state size, constraint count, ambiguity level, tool diversity, and recovery distance. For builders, this framework provides a practical checklist: if your task involves more than 5 branches, more than 3 dependent steps, or more than 2 consecutive failure recovery hops, you are operating in the zone where even frontier models drop below 50% reliability. The takeaway is clear: design your agent workflows to stay within these boundaries, or invest heavily in scaffolding and fallback logic.
Domain-Level Performance Variations
OmniaBench reveals striking differences across application domains. Models perform best in consumer-facing tasks like content generation, simple information retrieval, and basic form filling, where they achieve 65-75% Pass@1. Performance drops significantly in enterprise domains like compliance review, multi-system data reconciliation, and cross-platform workflow coordination, where scores fall to 35-45% even for the best models. Business domains like CRM data entry, invoice processing, and meeting scheduling land in the middle at 50-60%.
These variations matter for product decisions in a concrete way. If you are building a consumer AI assistant that helps users draft emails or summarize articles, the model layer may already be good enough to ship. But if you are building an enterprise agent that reconciles data across Salesforce, NetSuite, and a custom database, you are asking the model to operate in a domain where it fails more than half the time. That does not mean the product is impossible to build; it means you need to invest in guardrails, human-in-the-loop checkpoints, and aggressive error recovery at the platform level rather than relying on the model itself to handle edge cases gracefully.
What This Means for Builders
OmniaBench provides the most comprehensive capability map of frontier AI agents available today. The headline finding that even the best models score only 58% on general agent tasks is not a reason to abandon agentic products. It is a design specification for the infrastructure layer. If your product depends on reliable agent execution for multi-step tasks, you need to budget for three things that the benchmark makes visible.
First, invest in pre-flight validation that checks whether a task falls into the failure-prone zone before the agent executes it. Second, build robust guardrails that detect when an agent has violated constraints or made an unrecoverable error, and route to fallback paths rather than compounding the failure. Third, measure your own task-level success rates against the OmniaBench capability dimensions, so you know exactly which failure modes your stack handles well and which ones need work. The benchmark's ten-dimensional taxonomy and eight difficulty factors give you a vocabulary for this analysis that no single aggregate score can provide.
The paper is available on arXiv at arxiv.org/abs/2607.14989. The benchmark dataset contains 1,431 tasks with a reduced 644-task subset designed for lower-cost evaluation and reduced contamination risk. For teams building agentic products in 2026, this is the most practical reference point available for understanding what your model layer can and cannot do.