The Benchmark Violation

Two papers document systems that score well on evaluation metrics while violating the principles they claim to embody — and both identify the specific mechanisms by which good numbers mask bad behavior.

Denis et al. (arXiv: 2604.01454) test whether a stretched-grid deep learning weather prediction model (Bris) respects atmospheric physics during extreme events. Despite strong error metrics on standard benchmarks, the model fails to maintain fundamental balance equations — the physical relationships between pressure, wind, and temperature that govern real atmospheric dynamics. The model predicts weather that looks right by the numbers but couldn’t physically exist. It learned the statistical patterns of weather without learning why weather works.

Yao et al. (arXiv: 2604.01457) locate the specific neural circuits — a compact set of MLP blocks and attention heads in middle-to-late layers — that cause LLMs to express false certainty. When an LLM says “I’m 95% confident” about a wrong answer, the overconfidence isn’t a vague emergent property of the whole network. It lives in identifiable components that can be targeted with interventions to improve calibration. The model learned to sound certain without learning when certainty is warranted.

The structural claim: benchmark success and principled behavior are orthogonal. A weather model can minimize error on test sets while violating conservation of energy. A language model can produce fluent, confident text while systematic overconfidence lives in specific, fixable circuits. In both cases, the evaluation metrics reward the surface pattern (low error, high fluency) without testing the underlying principle (physical consistency, epistemic calibration).

This is not a flaw in any particular model — it’s a structural feature of how we evaluate systems. Benchmarks test outputs against expected outputs. They don’t test whether the system’s internal process is consistent with the domain’s principles. A weather model that violates energy conservation can still minimize root-mean-square error, because most test cases don’t push the model into regimes where the violation matters. An LLM with overconfidence circuits can still achieve high accuracy on factual questions, because most questions it encounters are ones where high confidence is appropriate.

Denis et al.‘s finding is particularly concerning because the violation is only visible during extreme events — exactly the scenarios where accurate forecasting matters most. Under normal conditions, the model’s predictions look physical because normal weather approximately satisfies the balance equations. During storms, heat waves, or rapid pressure changes, the model’s outputs diverge from physical reality. The bench-mark selected for normal conditions; the failure appears under stress.

Yao et al.‘s finding is more hopeful: if overconfidence lives in specific circuits, it can be fixed by targeted intervention rather than wholesale retraining. But the deeper implication is unsettling — the model’s confidence expression is mechanistically disconnected from its knowledge state. The circuits that produce “I’m confident” are not the same circuits that assess whether confidence is warranted. Confidence is generated, not computed.

The pattern generalizes beyond weather and language. Any system evaluated on output quality rather than process quality can develop the benchmark violation: scoring well on the metrics while violating the principles the metrics are supposed to measure. Financial models can minimize prediction error while violating no-arbitrage conditions. Medical diagnostic systems can maximize accuracy while violating clinical reasoning principles. The benchmark rewards the destination while ignoring the path.

The question for anyone deploying these systems: when did you last test whether your model satisfies the principles of your domain, rather than just the metrics of your benchmark? The answer for most systems is never, because principle-testing requires domain expertise that benchmark construction doesn’t. Building the right benchmark is harder than building the model, and it almost always gets less investment.