GAUGE finds 57.5% of agent chats rated satisfying still failed the user's task — Ranking validity vs construct validity — What does it mean?

The news. On 10 September 2026, Umesh Bodhwani, Thanh Tran and Kai Wei published GAUGE, an offline evaluation protocol aimed at the release gate most agent teams now run: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. Across 25 agents from six providers on the τ²-bench and SimulatorArena benchmarks, the authors compared that gate against a grounded verifiable reward and reported two separate findings — one about what the gate measures, one about how finely it can measure it. Read the paper →

Picture the examiner again. She rides the whole way, and at the end she writes down how the trip felt: smooth, calm, pleasant, nine out of ten. That score is not fake. She is careful, other examiners who ride the same route agree with her, and if you asked a panel of them to rank ten drivers by pleasantness they would broadly agree on the order. It is also completely silent on whether the car reached the address the passenger asked for.

Two different questions are hiding inside "is my eval gate any good?", and release practice usually only asks the first. Ranking validity asks whether the gate puts the candidates in the right order — did the better agent get the higher score? Construct validity asks whether the quantity the gate measures is the quantity you care about — does "pleasant" have anything to do with "arrived"? These come apart, because agreeing with humans is evidence about the first and no evidence at all about the second. A room full of examiners can be perfectly consistent with each other about pleasantness and perfectly uninformative about destinations.

That gap has a shape you can look for in your own pipeline. Teams validate a judge the way you would validate any annotator — sample some transcripts, check that people agree with its scores — and then read the resulting number as if it meant the task worked. Nothing in that procedure ever touches an outcome. This is the difference between scoring an agent and passing or failing it, and it is one of the four ways an eval quietly stops telling you the truth.

The first finding is the one that stings. GAUGE took the conversations a blind panel had rated satisfied and asked, separately, whether the customer's task had actually been completed. 57.5% of the conversations a panel called satisfying had failed the task. Satisfaction carried essentially no information about success — the two were decorrelated, and the pattern held across five rater populations, across both benchmarks, and across every subjective dimension the authors rated. That last clause is what closes the escape hatch: this is not a badly-worded rubric that a better prompt would fix. Every subjective dimension they tried behaved the same way, because the problem is not which opinion you ask for. It is that you are asking for an opinion about a conversation when what you want to know is what happened to the customer's account.

The second finding is subtler, and it is the reason the first one is dangerous rather than merely embarrassing. The gate's ordering is not junk. Over the broad capability span — a weak agent against a strong one — it agrees with the grounded reward on which should be promoted more than 99% of the time. Disagreement only starts climbing when the two candidates are close, and there it reaches 31%. The examiner can still tell a learner from a veteran. Put two veterans in front of her and pleasantness becomes noise.

So the gate keeps its resolution exactly where you do not need it and loses it exactly where you do. You rarely run a release gate to find out whether last year's model beats this year's flagship. You run it on this week's prompt change against last week's, on a candidate that is a hair better or a hair worse — which is the close-pair regime, the one where the gate is a coin weighted 69/31. And the failure is invisible from inside: a close pair produces a confident score difference just like a wide pair does. Nothing in the output tells you which regime you are in. That is the case for keeping an online signal alongside the offline one rather than treating the offline gate as the decision.

Put numbers on it. Say you ship twelve prompt-or-model changes a year and gate every one of them on the simulated-user judge. Four are coarse — a genuine capability jump, the kind the gate reads reliably — and eight are the ordinary close calls that make up most of a release calendar (illustrative split; the paper supplies the disagreement rates, not the calendar). On the four coarse ones the gate is wrong under 1% of the time: 4 × 0.01 = 0.04 bad promotions, call it zero. On the eight close ones it is wrong 31% of the time: 8 × 0.31 = 2.5 bad promotions. Same judge, same rubric, same year — about 2.5 of your 12 releases promote the worse agent, and every one of them comes from the half of the calendar where you were leaning on the gate hardest. Raise the close-call share from eight to ten and it climbs to 3.1; the number is driven almost entirely by how many of your decisions are genuinely marginal, which on a mature product tends to be most of them.

GAUGE's remedy is not "stop using judges." The authors propose a cadence they call calibrate-then-trust: measure the gate against a grounded verifiable reward before you let it decide anything. What the 1%-to-31% split then implies in practice — the paper names the cadence rather than prescribing a threshold — is that the calibration should tell you the reward gap below which its decisions stop tracking, so you can confine it to pairs wider than that. The gate keeps its job; it just gets a documented range instead of blanket authority.

The cheap half of that you can do this week. The authors propose a judge-free completion bit — a single yes/no on whether the run finished the task rather than getting cut off — as a zero-cost tripwire for truncation regressions. It needs no model and no rubric, which is precisely why it can referee one: a change that quietly starts truncating conversations can raise a satisfaction score rather than lower it, plausibly because a short, clean exchange reads better than a long, messy one that actually solved the problem. Calibrate first, then trust — and only inside the range where you measured the agreement. That belongs in the rollout gate itself, next to whatever canary you already run.

Goes deeper in: AI Agents → Evals & Diagnostics → Pass/Fail vs Score

Related explainers

• LLM judge as advisor — the structural fix for a mis-anchored judge: keep it, but put a deterministic gate in front of the decision.
• Trajectory-aware vs outcome-only grading — what a judge sees when you hand it the whole run instead of the final answer.
• Steerable bias directions — the other way a judge misleads you, from inside its own activations.