Every AI security agent on the market today claims a benchmark score. It solved 94 percent of penetration testing challenges. It caught 90 percent of simulated intrusions. But those numbers come from letting the agent run for as long as it wants, spend as much compute as it needs, and call as many tools as it likes. In real security operations, that is a luxury no team has. A new paper from researchers at Google and Robust Intelligence titled Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents introduces a framework that aligns AI security evaluation with operational reality. And the findings upend assumptions about which models are actually useful.
What the Paper Proposes
The core innovation is simple: instead of reporting only peak capability, evaluate models at fixed cost levels and decompose performance by inference spend and tool spend. The authors introduce cost-success curves as a standard metric, arguing that operational security depends on what an agent can achieve within a real world budget, not what it can do with unlimited compute. A model that solves 90 percent of challenges at $2 per attempt is more valuable than one that solves 95 percent at $20 per attempt, if the extra 5 percent comes from spending ten times longer on each challenge.
The framework evaluates agents on two distinct benchmarks. On the offensive side, they use Cybench, a set of 117 capture-the-flag cybersecurity challenges. On the defensive side, they use Splunk BOTS v1, a SOC investigation dataset where agents must navigate telemetry, query logs, and identify intrusions. By measuring both at fixed cost budgets, the paper surfaces a critical finding: offensive and defensive tasks scale in fundamentally different ways.
Offense Scales with Spend. Defense Does Not.
The results reveal two distinct scaling regimes. Offensive CTF performance improves smoothly with additional test-time compute. More reasoning steps, more tool calls, more budget consistently produce better results. GPT-5.5 leads the offensive benchmark at 94.1 percent Cybench pass@1, but no frontier model saturates within its run budget. Raising the per-sample dollar allocation keeps buying more solved challenges. For security teams evaluating red-team agents, this means that budget directly determines capability. There is no ceiling yet.
Defensive SOC investigation does not work the same way. The best defender, Claude Opus 4.8 at 93.8 percent of BOTS v1 points, reaches near its maximum effectiveness within roughly 20 tool calls per question. Additional calls and additional spend buy almost nothing. The model that leads defense is not the same model that leads offense. GPT-5.5, the offensive leader, scores only 81.0 percent on the defensive benchmark. Performance rank flips entirely depending on which side of the security operation you are measuring.
This is the paper's most operationally significant finding. A CISO evaluating AI security tools cannot look at a single benchmark score. The model that best automates penetration testing is not the model that best investigates a compromised network. And the model that excels at investigation does not benefit from simply being given more compute or more time. It benefits from disciplined tool use, efficient telemetry navigation, and selective enrichment of signals. Those are product design choices, not model capability questions.
Refusals Now Gate Offensive Capability
The paper also documents a surprising finding about safety alignment and offensive benchmarks. Claude Fable 5, Anthropic's most recent frontier model, refuses all 117 Cybench sample-epochs, scoring 0 percent on offensive challenges while achieving 88.4 percent on defensive tasks. GPT-5.6 Sol refuses 90.6 percent of offensive samples. These are not capability failures: the models are capable of solving these challenges, but their safety alignment policies prevent them from doing so. Refusal behavior, not raw ability, now determines offensive security benchmark scores for the newest models.
This has practical implications. A security team evaluating AI agents for offensive cybersecurity cannot rely on benchmarks from frontier models that simply refuse to participate. The paper suggests that specialized, fine-tuned models or models with adjusted safety thresholds may be necessary for red-team applications. It also means that defensive benchmarks, where refusal rates are lower, may be a more reliable measure of capability for aligned frontier models.
What This Means for Builders
For founders building AI-powered security products, this paper changes how you should think about evaluation. Three implications stand out. First, if you are selling an AI security agent, benchmark scores that do not include cost metrics are misleading. Any CISO who reads this paper will start asking: what can your agent achieve in 30 seconds versus 5 minutes, and what does each attempt cost? You need to be ready with those numbers.
Second, the finding that offense and defense scale differently means that a general-purpose agent is unlikely to excel at both. A product that claims to handle penetration testing, SOC investigation, vulnerability assessment, and compliance auditing with a single model may be sacrificing performance in every category. Specialized agents for specific security functions may win the market over generalists.
Third, refusal rates on offensive benchmarks are a critical product decision, not just a safety metric. If your target buyer is a red team, you need a model that will actually execute offensive tasks. If your target buyer is a SOC team, safety alignment that prevents offensive operations may be a feature, not a bug. The paper makes it clear that security product builders cannot ignore this trade-off.
The interactive results website at evals.frontier.security lets you explore every model's cost-success curve across both benchmarks. It is worth the visit. The data is transparent, the methodology is sound, and the implications for how we judge AI security agents will last longer than any single model's benchmark score.
