Paper Brief - 'Explainability Research Must Prioritize Foundations over Ad-hoc Methods': Top Researchers Argue the XAI Field Has Been Going in the Wrong Direction

How many times have you clicked a 'Why did this happen?' button in an AI product and received an explanation that changed nothing about what you did next? A new position paper from ten leading researchers argues this experience is not a bug but a feature of how the entire field of Explainable AI has been operating.

The paper, accepted to the ICML 2026 Position Paper Track and posted on arXiv (2607.14123), assembles a who's who of XAI research: Cynthia Rudin (Duke), Jennifer Wortman Vaughan (Microsoft Research), Himabindu Lakkaraju (Harvard), and seven other senior researchers spanning academic institutions and industry labs. Their central claim is blunt: despite the explosion of XAI techniques from feature attributions like SHAP and LIME to sparse autoencoders and mechanistic interpretability, explanations 'rarely influence real-world workflows.' In practice, they write, explanations are 'generated and discarded without guiding meaningful action.'

The paper is not a technical contribution but a disciplinary intervention. It calls on the machine learning community to stop inventing ever more ad-hoc explanation methods and start building the foundational infrastructure needed to make explanations actually useful.

The Four Structural Failures of Current XAI

The authors conduct a systematic analysis of recent ICML, NeurIPS, and ICLR papers and supplement it with a survey of XAI practitioners. They identify four recurring structural problems that, in their view, prevent the field from making cumulative progress.

The first is the absence of standardized evaluation frameworks for explanation usefulness. Most papers propose a new method and evaluate it on metrics like faithfulness or completeness that have weak correlation with whether the explanation helps a human make a better decision. The field has no agreed-upon benchmark for what counts as a useful explanation, which means every paper defines success differently and results cannot be compared across methods.

The second failure is the disconnect between explanations and downstream decision-making. Current XAI techniques tell you why a model predicted what it did, but they do not tell you what to do differently. A doctor reviewing an AI diagnosis that says 'because of feature X, Y, and Z' still has to figure out whether to trust it, override it, or investigate further. The explanation itself provides no action guidance, and no framework exists for connecting explanations to decision outcomes.

Third, the field suffers from over-reliance on toy benchmarks that do not reflect real-world deployment conditions. A method that works perfectly on a UCI dataset with 20 features and balanced classes may fail entirely when deployed in a hospital, bank, or courtroom where data is noisy, concepts are ambiguous, and users have varying levels of expertise. The paper argues that the community has systematically avoided the harder work of evaluating XAI in realistic settings.

Fourth, there is insufficient research on how different user groups actually consume explanations. A domain expert reading an explanation has different needs, background knowledge, and cognitive patterns than a general user or a developer debugging a model. Current XAI research largely treats 'the user' as a homogeneous entity, which the authors argue is one reason explanations fail to translate into action across diverse deployment contexts.

Why This Paper Matters Right Now

The timing of this intervention is significant. XAI is no longer an academic subfield. The EU AI Act requires explainability for high-risk AI systems. Enterprise procurement RFPs increasingly demand model interpretability. AI products ship 'why did this happen?' buttons as standard features. The market is demanding explanations, and the research community is supplying them, but as the paper documents, neither side has validated that the explanations being generated actually help anyone.

For founders building AI products for enterprises, this creates both a warning and an opportunity. The warning is that shipping explainability features without measuring their impact on user decisions is likely wasting engineering effort. If your product has a transparency feature that nobody uses or that does not change behavior, it is not serving its purpose. The opportunity is that building explanation systems that demonstrably improve decision quality would be a genuine competitive differentiator. The paper essentially hands you the product spec: design explanations that connect to actions, measure their impact, and iterate based on feedback.

The paper specifically criticizes the trend of proposing new XAI methods without demonstrating they lead to better decisions. This critique applies not just to academic papers but to any AI product shipping explainability as a checkbox feature. If you cannot point to data showing your explanations change user behavior, the paper's argument suggests you may be building the wrong thing.

What This Means for Builders

The paper is a position piece, not a solution, but its diagnosis points toward a concrete design philosophy for AI products. Three implications stand out for founders and developers.

First, the next generation of AI products should be explainability-native, not explainability-bolted-on. If you are building an AI copilot, decision-support tool, or automated reasoning system, the architecture should assume that users will need to question, challenge, and iterate on AI outputs as part of the core workflow. Explanations should not be a separate feature accessed through a modal button; they should be woven into the interaction loop.

Second, the paper's emphasis on evaluation creates a product opportunity. A startup that could offer 'explanation effectiveness testing' as a service or a standardized benchmark connecting explanations to decision outcomes would fill a gap the paper identifies at the disciplinary level. Enterprises deploying AI in regulated industries need this infrastructure today and have no good options for getting it.

Third, the paper implicitly asks whether the current obsession with mechanistic interpretability and sparse autoencoders which dominates XAI research discourse is also vulnerable to the same critique. Understanding what a model represents internally is interesting, but unless that understanding leads to better decisions, better audits, or better debugging, it may be another ad-hoc method in search of a problem. Builders should allocate their interpretability budget toward techniques that connect to their actual decision pipeline, not the ones that produce the prettiest visualizations.

The paper concludes with a practical checklist designed to shift XAI toward human-centered, action-oriented paradigms. For founders racing to ship explainability features ahead of regulatory deadlines, that checklist may be worth reading before the next sprint starts.