Hidden error in LLM pipelines leads to benchmark hacking and undercoverage
Paper SI Appendix R package Replication Follow
LLM evaluations drive model deployment, safety standards, and research. Yet we ignore noise from the prompt, the judge, the agent harness, and so on. So we underestimate error in benchmarking and in annotation, and the problem gets worse with more data.
So I argue in Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking. I’ll get to what this means for confidence intervals and how to fix it. But first, let’s talk about how this makes benchmark hacking easier.
Benchmark hacking
Benchmark hacking (like p-hacking) is where you test many versions of a model with slightly different setups and only publish the best score. It depicts noise as signal.

Are labs doing this? If so, on the most prized benchmarks we should see higher scores on their own runs than from independent evaluators. Indeed, on SWE-bench Verified, labs’ launch scores beat independent runs by 4 to 5 points on average. Unique harness and other settings likely drive much of the difference. Anthropic’s Sonnet 4.5 and Haiku 4.5 scores, for example, used a “minor prompt addition” telling the model to “use tools as much as possible, ideally more than 100 times” and to write its own tests first. Anthropic’s Sonnet 4.5 run scored 6 points above Epoch’s run. To be clear Anthropic disclosed this, and later launches report prompt-modified scores separately. (This analysis and the next are meant to tie this to the real world and are not in the paper.)

More evidence comes from Terminal-Bench 2.0 data, another high-value benchmark. When labs used a different agent harness and settings, like Codex or Claude Code, they reported scores 7 to 19 points above the same model on the benchmark’s default harness, Terminus 2. That’s for the six models with a Terminus 2 run on the leaderboard. Note also that in runs that Alibaba’s Qwen team published of other labs’ models, tool-call format moved scores by up to 11 points, and which format did better depended on the model.

Is using a better harness really “noise?” Well, scores depend on the harness, and different harnesses help different models. So if we’re trying to measure capability and we ignore the harness there’s hidden variability.
Benchmark hacking is a different problem from “benchmaxxing,” which mostly means post-training on tasks that mimic a benchmark.
More about benchmarks and benchmaxxing
Evaluators do try to combat other problems, such as test items leaking into training data or agents gaming the grader. To deal with test items in the training data, SWE-Bench Pro holds back 858 of its tasks and takes 276 more from private company codebases, and SWE-rebench keeps scoring models on recent GitHub tasks and labels results that may be contaminated.
Other countermeasures focus on actions agents take during an eval run. For example, on SWE-bench, agents fix real GitHub issues. In the past, sometimes the very GitHub repo contained the bug fix, which agents promptly copied (SWE-bench issue #465). SWE-bench Pro V2 builds its evals without these answers. The benchmark sandbox also generally blocks internet access so the agent can’t look it up online either. In Terminal-Bench 4.1, they took measures to prevent reward hacking, and the leaderboard’s integrity rules call for an agent judge to review every trial an agent passes.
To help prevent benchmark contamination, ARC Prize tests models on its own semi-private eval set and reports every run so the lab can’t simply pick the best. ARC calls the set semi-private because its tasks pass through APIs, which our double-blind evaluation system will go a long way toward solving. Epoch AI, Vals.ai, and the SWE-bench and Terminal-Bench teams also rerun models under their own setups, but that usually comes after the lab has published its benchmark number. Before launch, there’s little to stop a lab from trying many harnesses, prompts, or compute settings and reporting the best.
Research using LLM judges for annotation (autoraters) has the same problem. Baumann, Röttger, and colleagues (2025) re-evaluated 37 LLM annotation tasks from 21 studies. They found that 31% of hypothesis tests on LLM-labeled data reached wrong conclusions under other models and prompts, and that adversarial choices could flip the sign of 68% of significant effects.
Why naive error bars get worse with more data
Current practice is to report confidence intervals (CIs) that treat items as a sample but fix the prompt, judge, and harness. We should expect under-coverage to worsen with more items, because omitted variance from prompt and judge choices does not shrink with N.

In the paper’s simulation, as we add items, the usual interval (A) contains the truth less often: 42% of the time at 25 items and 11% at 400. Once we add the omitted prompt, judge, and temperature variance, coverage is back above 95%.

By the way, sometimes you hear “Just set the temperature to zero,” which does NOT fix this. Doing so picks the single most likely token at each step (greedy decoding), which simply further hides the variation that sampling would have shown. Note that infrastructure nondeterminism alone has produced standard deviations of up to nine percentage points in task accuracy at temperature zero (Yuan et al. 2025).
What undercoverage looks like on a leaderboard
We see this play out on Terminal-Bench 4.0, where I found data for Vals.ai and the official board runs on the same 66 tasks on the same 39 models. The tasks are the same so the difference between a model’s scores on the two runs comes from rerun noise plus differences in how each evaluator ran the model. Both boards’ error bars count only reruns. If they captured all the noise, about 5% of pairs would differ by more than their 95% intervals allow. 21% do (8 of 39). This analysis is also not in the paper.

Once we put reasonable CIs on benchmarks, we can see how silly it is when we fight about a 2-point lead. This is the top of Vals.ai’s Terminal-Bench 4.0 board. Grey are CIs based on rerun noise, which is what the leaderboard reports. Red also accounts for which tasks were picked and who runs the model. That’s not ideal, but it’s better. Under the better intervals, fully 10 models tie with the leader!

These intervals do not account for everything. They leave out each harness’s average shift (one harness runs every model on Vals.ai’s board, so it leaves the ranks unchanged), the checker version (Terminal-Bench 4.1’s fixed checkers moved official scores by -1.2 to +2.7 points, and Vals.ai’s board is 4.0), and rewording of each task’s instructions.
The paper finds the same on Terminal-Bench 2.0. Its authors ran 17 models across six harnesses, about 5 trials per task, so we can estimate how much a model’s score depends on the harness beyond rerun noise: 3.5 points (SD). Of the 16 neighboring pairs of models run under Terminus 2, the published intervals separate 4. Once harness variance is added, none separate.
Many leaderboards fall short of even the naive interval. SWE-bench and τ-bench report no interval at all. MLE-bench, PaperBench, Vals.ai, and Terminal-Bench report intervals from reruns alone, which hold the tasks fixed. OpenAI’s system cards say their own resampling approach “can lead to overly tight confidence intervals.”
If you run a leaderboard, please show the red CI. These intervals account for which tasks were picked and how each model was run, along with reruns. To pilot, run a few models under 2-3 harnesses or prompts.
Total Evaluation Error
To get reasonable estimates of that noise, the paper introduces Total Evaluation Error (TEE). The name follows total survey error, which treats every stage of a measurement pipeline as a source of error. TEE uses a mixed model to split the variance among items, prompts, judges, temperatures, reruns, and their interactions. It builds on generalizability theory and uses its standard estimators. For agents, the harness sets the system prompt, the tools, and the agent loop, so different harnesses count as different prompt variants.

What we found
Judges dominate safety ratings. On AILuminate v1.0 safety prompts from MLCommons, scored by a panel of three LLM judges, judge-related terms account for 73% of the variance! In our safety and ideology applications, naive standard errors are 53 - 70% smaller than the corrected SE.

Averaging lowers the error itself. A small pilot can show you where error comes from, and hence what to average over. On AILuminate, scoring with 3 prompts and 3 judges instead of one reduces the score’s error by about half, for 9× the API calls. That’s a much better allocation than spending API calls on replicates from the same model and prompt. Running the full factorial design takes 120× the calls, for only an extra 5% reduction. In the paper’s SI, a small pilot recovers the qualitative design guidance of a full run 8 times larger.

More data, worse coverage. On Chatbot Arena matches scored by LLM judges, naive 95% CIs from one judge and one prompt cover the 3-judge × 5-prompt average reasonably well at 100 matches, but only 78% of the time at 2,000, while TEE’s stay at or above 95%. More data narrows the interval around one configuration’s average, so coverage gets worse.

Scores get closer to ground truth. On MMLU, when we spend the same API budget where TEE says, scores at larger budgets end up much closer to true accuracy than with the standard approach.

What to do about it
So what to do? In our paper applications, when a pipeline has an LLM judge, averaging across multiple judges reduces error more than any other single change. Don’t rest results on one judge model or prompt template. For agent leaderboards, run models under more than one harness. To help ensure generalizability, once you have variation over judge and prompt/harness, you can sample more items. And run a small pilot first – I suggest at least 30 items, 3 prompts, 2 judges, and 3 replications, or about 540 API calls in the paper. The R package fits a decomposition from pilot data, projects standard errors under other designs, and recommends budget allocations.
What TEE can’t tell you
TEE focuses on reliability. You need gold standard data to understand how often a judge is correct, whether you have a valid construct, and whether the items actually cover that construct. If you don’t have coded hate speech (dog whistles), your toxicity benchmark will give precise but incomplete estimates. The framework also doesn’t account for variance from reasoning traces, and my demonstrations are English-only with models served via API.
How this paper came about
This work started with our Nature paper on state media and LLMs, see also (project site), with Hannah Waight, Eddie Yang, Yin Yuan, Molly Roberts, Brandon Stewart, and Josh Tucker. I looked for a framework like this but couldn’t find one. The confidence intervals in that work do not use TEE, though the website already uses multiple judges as TEE recommends, and I’ll update the website soon.
Peter Mattson at MLCommons asked me to think about this and try to figure out a framework for AI benchmarks, starting with their safety benchmark AILuminate, which became the paper’s first example. Thanks to MLCommons for supporting this work, along with NYU’s Center for Social Media, AI, and Politics, particularly Josh Tucker and Jonathan Nagler. Thanks also to participants at the 2026 CSMAP conference for helpful feedback. And thanks for the feedback and early conversations that shaped this paper to Chris Barrie, Ben Guinaudeau, Aaron Kaufman, Brandon Stewart, Kylan Rutherford, Bennett Hillenbrand, Andrew Gruen, Rebecca Weiss, Hannah Waight, Molly Roberts, Eddie Yang, Melina Much, Joe Higton, and Jennifer Allen.
On using AI
This work would not have happened without Claude Code (Claude Opus 4.6, 4.7, 5, and 5.5) to help:
- massively expand the literature search
- multiple round of critical review on initial drafts of the paper, suggesting improvements
- thinking through additional sources of potential variation in LLM pipelines
- search for and collect extant data that helps demonstrate the problem in the real world, including the leaderboard data in this post
- draft and recheck the code for data collection, analysis, simulations, figures, and the R package
- check every number in the paper against code output multiple times
- fact-check claims against primary sources
- cleanly move the content between journal/venue formats, including Neurips poster and slides
- draft the 55-page SI Appendix, TiKZ diagrams, LaTeX math, etc.
Paper: arXiv 2604.11581 (SI Appendix)
R package: github.com/SolomonMg/totalevalerror
Replication files: github.com/SolomonMg/totalevalerror-replication
Come see the poster in Sydney: Wednesday, December 9, 10 AM - 1 PM AEDT, Hall 1-4.