On July 17, a Kaggle user named twerkmeister published a detailed analysis of what they called 'inconsistencies in the evaluation process and selection of winners' for Kaggle's flagship Measuring Progress Toward AGI Cognitive Abilities competition. The post rocketed to the top of Hacker News within hours, drawing 428 points and 268 comments. It also surfaced a question the AI industry has been dancing around for months. When both the contestants and the judges are AI systems, who is actually winning anything of meaning?
What the Controversy Reveals
Why it matters: Kaggle is the world's largest data science competition platform, now owned by Google. It sets de facto standards for what counts as meaningful progress in artificial intelligence. If its evaluation pipeline is compromised by the same AI slop it is trying to measure, every benchmark built on top of it inherits that rot. For founders building products on AI models, the performance numbers they use to choose vendors might rest on broken ground.
The Measuring Progress Toward AGI competition was designed to do exactly what its name suggests: test how close modern AI systems are to general intelligence by evaluating their cognitive abilities. Instead, the discussion thread turned into a full indictment of the entire competition ecosystem. One HN commenter named ecshafer captured the heart of the frustration. 'Kaggle is most likely using AI to assess the submissions and are not using any common sense by blindly accepting the results,' they wrote.
The deeper structural problem is what happens when LLM-as-a-judge becomes standard practice. The evaluation pipeline becomes a closed feedback loop. AI generates submissions. AI evaluates those submissions. AI ranks the winners. At no point does a human actually read the output and say 'this makes sense.' One HN commenter noted that major ML and AI conferences are being inundated with AI-generated papers, and the review system has no reliable way to filter them out. What starts in competitions inevitably spreads to peer review, hiring pipelines, and product benchmarks.
The Kaggle Measuring Progress Toward AGI competition faced credibility issues when a user posted evidence of evaluation inconsistencies. The post hit #1 on Hacker News with 428 points and 268 comments. The controversy exposes a broader crisis: LLM-as-a-judge creates a closed feedback loop where AI generates, evaluates, and ranks submissions with minimal human oversight. The MEDLEY-BENCH paper (arXiv:2604.16009) found a systematic knowing/doing gap across all 35 models tested, suggesting AI systems are better at appearing competent than actually being competent. Conference organizers and platform operators are struggling to handle AI-generated submissions that crowd out human work, with no scalable solution in sight. Smaller, cheaper models often matched or outperformed larger counterparts on metacognitive benchmarks, challenging the assumption that scale equals capability.The Deeper Problem With AI Judges
The timing of this controversy is especially awkward for Kaggle and Google. Just weeks before the HN post blew up, researchers published MEDLEY-BENCH on arXiv. The paper, led by Farhad Abtahi and colleagues, evaluated 35 models from 12 families on 130 ambiguous instances across five domains. Its central finding: a robust evaluation/control dissociation. Evaluation ability increases with model size within families, but control does not. In every single one of the 35 models tested, evaluation was the weakest relative ability, indicating a systematic knowing/doing gap.
In other words, bigger models look smarter. They can evaluate and critique. But they cannot reliably act on that evaluation. They know what the right answer is but still produce the wrong output. The MEDLEY-BENCH authors describe this as a 'scale buys evaluation but not control' problem, and it maps directly onto what the Kaggle critics are seeing. Submissions that score well on automated evaluation metrics may still be fundamentally unreliable when deployed in the real world.
Why This Matters for Founders
The Kaggle discussion also linked to an arXiv paper on AI metacognition benchmarks that explores similar territory. That paper, General Scales Unlock AI Evaluation with Explanatory and Predictive Power, examined how metacognition and reasoning are affected by model size, chain-of-thought prompting, and distillation. It found that high predictive power at the instance level is possible using demand scales, but that even the best evaluation frameworks struggle to separate genuine capability from sophisticated mimicry.
The HN thread was not just about Kaggle. Commenters connected the controversy to a wider pattern. AI-generated papers flooding peer review. Automated screening systems that hallucinate reviewer feedback. Companies shipping product benchmarks that nobody outside the company has audited. One commenter pointed out that the incentives are all wrong. Competition organizers want high engagement and flashy leaderboards. Participants want to win. Neither party has strong motivation to prove the evaluation is actually measuring what it claims to measure.
Another commenter named jagged-chisel zoomed out further, arguing that the root cause is cultural rather than technical. 'I think we need to address the underlying causes of people outsourcing their thinking like that. And a big contribution is move fast. No one has time to read, process, and think, because The Powers That Be (capital) want their results now.' This captures the uncomfortable truth: the Kaggle controversy is not an isolated incident. It is a symptom of an industry that has optimized for speed and scale at the expense of rigor and verification.
The MEDLEY-BENCH paper offers one hint of a way forward. Its most striking finding was that smaller and cheaper models often matched or outperformed larger counterparts on metacognitive measures. This suggests that metacognitive competence is not simply a function of scale. Training that rewards calibrated, proportional updating rather than raw output quality could produce models that actually know what they do not know. But implementing that at the competition and benchmark level would require Kaggle and similar platforms to rethink their entire evaluation philosophy.
PLUS: The MEDLEY-BENCH paper also introduced two complementary scores: the Medley Metacognition Score (MMS), a tier-based aggregate of reflective updating, social robustness, and epistemic articulation, and the Medley Ability Score (MAS), derived from four metacognitive sub-abilities. Under within-model relative profiling every single one of the 35 models showed evaluation as its weakest relative ability. No model escaped the knowing/doing gap. Expect at least one major competition platform to announce an evaluation audit or a human-in-the-loop overhaul within the next 90 days. The pressure from the HN thread combined with the academic evidence from MEDLEY-BENCH creates a tipping point. Platforms that do not respond will face a credibility exodus as serious researchers and competitors move to venues they trust. Within 12 months, LLM-as-a-judge will carry a stigma similar to p-hacking in academic research: something that signals the results should be treated with deep skepticism rather than accepted at face value.