Most medical AI benchmarks ask a single question: did the model get the right answer? That framing assumes the problem is one of factual knowledge. But in real clinical settings, a model can produce a text-book-correct answer and still recommend something dangerously wrong. A patient presenting with chest pain might get a perfectly accurate explanation of cardiac risk factors, while the model fails to flag that this specific case needs immediate escalation to a human physician. The right answer to the wrong question is not just useless -- it is actively dangerous. A new open-source benchmark called MedFailBench, released on arXiv by clinician-researcher Goktug Ozkan under Apache-2.0 and CC-BY-4.0 licenses, reframes the entire problem. It does not ask whether the model answered correctly. It asks which safety boundary failed.

The Problem With Right/Wrong Benchmarks in Medicine

Current medical AI benchmarks like MedQA, MedMCQA, and PubMedQA test factual recall. They present a multiple-choice question drawn from medical exam content and ask the model to select the correct answer. These benchmarks serve a purpose: they tell you whether the model knows basic medical facts. But they tell you almost nothing about whether the model is safe to deploy in clinical workflows.

Consider a scenario that MedFailBench's author uses to motivate the benchmark: a patient sends a message through a healthcare portal describing symptoms that could indicate a pulmonary embolism. A factual-recall benchmark would test whether the model knows the risk factors for PE. MedFailBench tests whether the model identifies that this specific case requires urgent escalation to a clinician. Those are fundamentally different capabilities. A model can ace the first test and fail the second, and the second failure is the one that gets patients hurt.

This blind spot in existing benchmarks has real consequences. Multiple studies have documented cases where AI-powered clinical decision support tools missed critical safety signals because the underlying models were optimized for accuracy rather than safety awareness. The models knew the right answer but did not know when to stop relying on their own knowledge and hand off to a human. MedFailBench is designed to surface exactly this class of failures.

Six Safety Gates: A New Vocabulary for Medical AI Errors

Instead of a pass/fail label, MedFailBench classifies errors across six safety gate types, each annotated with a severity score from 1 to 5. The taxonomy was designed by clinicians who understand how AI failures manifest in actual medical workflows.

The first gate is missed urgent escalation. This is the pulmonary embolism scenario: the model fails to recognize that a case exceeds its scope and needs immediate human intervention. It is arguably the most dangerous failure mode for autonomous medical AI, because it means the system stays quiet when it should be sounding an alarm.

The second gate is unsafe remote dosing. A model recommends a medication dose that would be appropriate in a controlled clinical setting but is unsafe to administer remotely without in-person monitoring. Think of a model suggesting a sedative dosage for a patient who lives alone -- the dose might be correct by the formulary, but the context makes it unsafe.

The third gate is unsafe discharge reassurance. The model tells a patient they are fine to manage at home when the clinical picture warrants further observation. This is a variant of missed escalation but specific to the discharge decision point, which carries its own legal and ethical weight.

The fourth gate is evidence fabrication. The model invents citations, test results, or clinical evidence to support its recommendation. This is a hallucination category, but MedFailBench frames it as a safety boundary violation rather than a quality issue. Fabricated evidence that supports a correct recommendation is still a safety failure because it undermines the trust chain in clinical decision-making.

The fifth gate is unsafe protocol execution. The model follows a clinical protocol correctly, but the protocol itself should not have been applied in this specific case. This is a subtle but critical failure mode: the model does everything right by the book, but the book was the wrong book.

The sixth gate is source support gap. The model makes a recommendation that lacks adequate evidentiary support from available sources. This differs from fabrication in that the model is not inventing sources -- it is simply operating beyond the evidence base, which in a clinical context is a safety issue in itself.

How MedFailBench Works: Architecture and Design Decisions

The current release, v0.2.1, contains 44 clinician-reviewed synthetic cases. Each case was constructed by clinical experts to test specific safety boundaries, then validated by independent reviewers. Every case includes a ground-truth annotation specifying which safety gates were triggered and at what severity level.

The benchmark avoids patient data entirely. All cases are synthetic, meaning there are no HIPAA or privacy concerns in running evaluations. This is a deliberate design choice: the benchmark is meant to be widely accessible to researchers and startups who may not have access to clinical datasets.

MedFailBench ships with several supporting artifacts beyond the case set. There is a safety gate taxonomy that defines each failure mode with clinical examples. There is a clinical severity rubric that standardizes how severity is assigned. There is an automated pipeline for archiving model-response screening runs, so evaluation results are reproducible. A live HuggingFace leaderboard preview tracks how different models perform on the benchmark, though the current release explicitly does not include model rankings.

The Apache-2.0 license means startups and enterprise teams can integrate the benchmark into their evaluation pipelines without legal friction. The CC-BY-4.0 license on the taxonomy and rubric means clinicians and researchers can adapt and extend the classification system for their own use cases.

What This Means for Builders

If you are building medical AI -- a clinical decision support tool, a patient-facing chatbot, a remote monitoring system, or a diagnostic assistant -- MedFailBench is the first benchmark you should run against your model. Not because it replaces existing benchmarks, but because it tests something no existing benchmark tests: whether your model knows when to stop relying on itself.

The safety gate taxonomy is not just an evaluation tool. It is a design tool. Each gate maps to a specific architectural requirement. If your system could ever produce an unsafe remote dosing recommendation, you need a guardrail that intercepts that specific failure mode. MedFailBench gives you the vocabulary to specify those guardrails and the test cases to verify they work.

For early-stage startups that cannot afford clinical safety teams, MedFailBench fills a critical gap. A solo founder building a medical AI product can run their model through 44 clinician-validated safety scenarios and get structured feedback on where their system fails. That is the kind of capability that was previously available only to well-funded health-tech companies with access to clinical partners.

The regulatory implications are worth watching. The FDA and equivalent international bodies are developing frameworks for evaluating AI safety in clinical settings. MedFailBench's safety gate taxonomy aligns closely with emerging regulatory thinking around AI risk classification. Running your product through this benchmark now gives you a head start on the structured safety documentation that regulators will eventually require.

The benchmark is also designed to evolve. At 44 cases, v0.2.1 is a foundation, not a final product. The open-source model invites contributions from the clinical community. As more cases are added and more failure modes are identified, MedFailBench has the potential to become the standard reference for medical AI safety evaluation. The question for builders is not whether you will need to test against it. It is whether you start now, while there are 44 cases, or wait until the benchmark has grown to 400 cases and your competitors have already closed the safety gap.