Two findings fall out immediately (and neither were visible to a conventional eval).
First, the cliffs. This query scores a perfect F1 = 1.00 at neutral phrasing — and 0.00 at high ambiguity. It is the satellite-48445 case from above: drop the distinguishing token “TLE” and the agent loses the table entirely. Same query, same agent, same ground truth; one notch vaguer and it falls off a cliff. A static benchmark tests the neutral phrasing, stamps “solved,” and reports flat ground where there is a precipice. Pass/fail was particularly misleading in that it did not just miss the cliff, but it told us the terrain was level.
Second, the sweet spot. For Discovery Agent, medium ambiguity beat neutral, and low ambiguity sometimes underperformed it. More specificity is not monotonically better for the system being evaluated; there is an optimal amount of steering. That is a graded, actionable signal. This is the “how close, how hard” texture we were missing from a scalar. It tells you where to hill-climb, or improve, the agent: in our case, straight at concrete failure modes like time-sharded tables (precision collapsing to ~8% as the agent over-retrieves 21 near-identical shards for a two-table answer) and context blow-up (F1 dropping from 0.75 to 0.32 once a query triggers long search chains). The map did not just say that the agent failed, but it said where, and why. Note that our hypothesis that less ambiguity and more context (via steering terms) should improve retrieval generally holds true, but for the specific Discovery Agent being exercised, the idiosyncratic “sweet spot” meaningfully highlighted trade-offs in its implementation.
We’re not alone
The field is converging on meta-benchmarking and exerting greater control of how we challenge and evaluate our agents. A growing body of work uses item response theory, the latent-ability model behind standardized testing, to treat difficulty as a measured quantity rather than a label: tinyBenchmarks and metabench show that a handful of informative items reproduce a model’s full score, and PSN-IRT turns the same lens on benchmark quality itself. Others audit the ground truth directly: MMLU-Redux found that 6.49% of Massive Multitask Language Understanding (MMLU) questions are mislabeled, and Platinum Benchmarks re-cleaned ten datasets to minimize both label errors and ambiguity — the same two axes we sweep for. And ambiguity is increasingly treated as intrinsic rather than noise: AmbigQA showed that a large fraction of real questions admit multiple readings, and later work finds that apparent hallucinations often stem from query ambiguity rather than model failure. What we have not seen elsewhere is the combination: information-theoretic ambiguity sweeping applied as a meta-benchmark over live enterprise data.
A benchmark we trusted turned out to be broken
We built our first evaluation on kramabench-astronomy, a benchmark established in the field, and one which other teams had already leaned on for their own evals. Teams derived benchmarks from this dataset, and we hypothesized subtle issues may have been introduced over time. When we actually read the benchmarks used by teams, with Gemini’s help, we found it was wrong in meaningful ways: ground-truth tables that did not answer their query, a question whose 124 sharded tables exceeded what some teams’ retrieval APIs could even return, months specified where exact dates were required. Quietly broken ground truth means quietly wrong conclusions not just for us, but for every prior analysis built on it.
This is the generalized crux of the matter: an evaluation is itself an artifact that can be defective, and almost nobody evaluates it. We instrument the agent and trust the ruler, but where do we validate that the measuring stick makes sense?
When two maps disagree
Now the recursive turn: If difficulty is something we generate, then we need to evaluate the generator itself; we should not trust it blindly either.
So we built the same ambiguity sweep two ways: steering terms from a pure-LLM guess, versus terms grounded in TF-IDF surprisal. The two disagreed violently. At high ambiguity, the LLM-built sweep scored the agent at F1 ≈ 0.34; the grounded sweep, ≈ 0.85. One of these maps is badly distorted. The grounded one, predictably, is the more robust: surprisal gives it a footing the free-running LLM lacks.
This is “evaluate your evals,” made concrete. The information-theoretic lens does not only grade the agent along a continuous axis; it grades the benchmark’s own construction, and adjudicates between the two.
Evaluate your evals
We have spent years optimizing agents against rulers we never measured. The bitter irony is that better models make this worse: as agents clear coarse benchmarks, the score saturates near the top and the exam loses its ability to highlight where the agent can be improved.
So the call to action is uncomfortable and overdue: evaluate your evals. Read your ground truth. Treat difficulty as a measured quantity, not a label: sweep it, plot it, find the bit-width where your system breaks. Ask not just “did it pass?” but “how close was the miss, how hard was the pass, and would a slightly vaguer question have sent it off a cliff?” Build evaluations that produce signals; not just verdicts.
There is a genuine tension to sit with here. Difficulty-as-entropy is only as reliable as the model that estimates the entropy. There’s a risk that if we push too hard on a measurable proxy, we optimize the ruler instead of the agent. That is not a reason to retreat to pass/fail; it is a reason to keep the evaluator under the same scrutiny as what it is evaluating. The moment we stop asking who evaluates the evaluators is the moment our maps stop being useful again.
1. Maia Polo, F. et al. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML 2024. arxiv.org/abs/2402.14992
2. Kipnis, A. et al. metabench: A Sparse Benchmark of Reasoning and Knowledge in Large Language Models. ICLR 2025. arxiv.org/abs/2407.12844
3. Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory (PSN-IRT). AAAI 2026. arxiv.org/abs/2505.15055
4. Gema, A. P. et al. Are We Done with MMLU? (MMLU-Redux). 2024. arxiv.org/abs/2406.04127
5. Vendrow, J. et al. Do Large Language Model Benchmarks Test Reliability? (Platinum Benchmarks). 2025. arxiv.org/abs/2502.03461
6. White, C., Dooley, S. et al. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. 2024. arxiv.org/abs/2406.19314
7. Min, S. et al. AmbigQA: Answering Ambiguous Open-domain Questions. EMNLP 2020. aclanthology.org/2020.emnlp-main.466
8. Lai, E., Vitagliano, G. et al. KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes. 2025. arxiv.org/abs/2506.06541
