What is predictive validity for AI agent evals?

Predictive validity is a measurement-theory idea IBM applies to agent leaderboards: instead of ranking agents by their aggregate score, you measure the rank correlation between a benchmark's in-sample ranking and the ranking it produces out-of-distribution. A high correlation means the leaderboard predicts real-world ordering; a low correlation means the score is a poor guide to which agent to actually deploy.

Why are aggregate-score agent leaderboards misleading?

Because they collapse a whole agent into one mean number measured under a single distribution of tasks, then sort by it. IBM's 'Beyond Static Leaderboards' ran the same industrial-agent benchmark fourteen ways and found the rankings disagreed, and public-to-hidden competition retrospectives show the same rank instability. The sorted order looks authoritative but does not transfer once conditions shift, so it is a weak signal for deciding what to ship.

How does predictive validity relate to distribution shift?

Distribution shift is exactly the condition predictive validity tests. In-sample means the tasks the benchmark measured; out-of-distribution means anything different in deployment — new task types, a new orchestration, a shifted input mix. Predictive validity asks whether the agent ranking holds across that gap, and IBM structures it as a twelve-tier apparatus with three falsifiable out-of-distribution criteria so the claim can be checked rather than assumed.

Agent leaderboards mislead under distribution shift (IBM) — Predictive validity

TL;DR

What is it: A new IBM paper, "Beyond Static Leaderboards", argues that the way we rank AI agents is broken: a leaderboard collapses each agent into one aggregate score and sorts by it. The fix it proposes is predictive validity — the rank correlation between a benchmark's ranking and the ranking you'd see out-of-distribution.
Why it’s needed: A single leaderboard number is a weak signal for real deployment. The whole point of an eval is to tell you which agent to ship — and if the benchmark's #1 isn't your deployment's #1, the ranking you trusted pointed at the wrong agent. This is the core lesson of Evals & Diagnostics and Production Evals.
vs previous: Where the old way ranks agents by their aggregate mean score on one benchmark and trusts that order, predictive validity asks a sharper question: does that order survive a distribution shift? IBM's finding is blunt — aggregate-score rankings do not transfer out-of-distribution.

Jargon

Predictive validity: Borrowed from measurement theory: does a test's score predict the real-world outcome it claims to measure? For agent evals, IBM defines it as the rank correlation between in-sample and out-of-distribution results — not the raw score, but whether the ordering of agents holds up when conditions change.
Aggregate score: The single number a leaderboard reports per agent — typically a mean across many tasks. It is easy to sort by, but it throws away the variance that tells you whether the ranking is stable. See AI Agents → Evals & Diagnostics.
In-sample vs out-of-distribution (OOD): In-sample = the conditions the benchmark actually measured. Out-of-distribution = anything different in deployment — new task types, a new orchestration, a shifted input mix. The gap between them is where leaderboards quietly fail; production teams watch it as drift.
Rank correlation: A measure of how well two rankings of the same items agree — +1 is identical order, 0 is unrelated, −1 is reversed. Predictive validity is this number, computed between the in-sample and OOD rankings.
Rank instability: When a small change in conditions reshuffles the leaderboard — the agent ranked first in-sample lands third out-of-distribution. IBM points to public-to-hidden competition retrospectives as direct evidence this happens.
Falsifiable criterion: A pass/fail test you can actually fail. IBM frames predictive validity through three falsifiable out-of-distribution criteria, so a benchmark's claim to validity can be checked and rejected — not just asserted.
MCP-based agent benchmark: A benchmark built on the Model Context Protocol tool interface, so the same agent harness can be re-implemented many ways. IBM ran fourteen parallel implementations of one such industrial-agent benchmark.

The news. On June 18, 2026, an IBM-led team (Dhaval Patel et al.) posted Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents to arXiv. They ran fourteen parallel implementation studies of an MCP-based industrial-agent benchmark — varying asset classes, orchestrations, retrieval strategies, and reasoning modes — and aggregated seven prior agent benchmarks. The headline: rankings derived from aggregate scores do not transfer to out-of-distribution settings. In place of one number, they propose ranking benchmark configurations by predictive validity: the correlation between in-sample and out-of-distribution rank, structured as a twelve-tier measurement apparatus with three falsifiable criteria. Read the paper →

Picture timing a field of sprinters indoors, on a fast track with no wind, and printing the ranking from their personal bests. On paper you now know exactly who is fastest — first, second, third, in order. Then race day comes, outdoors, into a gusting headwind, and the podium reshuffles: the indoor record-holder fades to third, and someone who was never the fastest indoors wins the race that counts. The indoor clock wasn't lying — it measured real speed in one setting. It just had no way to tell you whether that order would survive the wind. The sprinter is an agent, the indoor ranking is an aggregate-score leaderboard, the outdoor race is deployment, and the question the indoor clock can't answer is predictive validity.

A leaderboard does exactly what the indoor clock does. It runs each agent over a fixed battery of tasks, averages the results into one aggregate score, and sorts. That sort is the product everyone consumes — the tweet, the ranking row, the "best open agent" headline. But the average is measured under one distribution of tasks, and IBM's central result is that the ordering it produces does not hold once the distribution moves. When they built the same industrial-agent benchmark fourteen different ways — swapping orchestrations, retrieval strategies, and reasoning modes — the rankings disagreed with each other, and public-to-hidden competition retrospectives showed the same rank instability in the wild.

The deeper move is to stop treating the benchmark as a scoreboard and start treating it as a measurement instrument — and to ask of any instrument the measurement-theory question: does its reading predict the thing you actually care about? IBM operationalizes that as predictive validity: the rank correlation between a configuration's in-sample ranking and its out-of-distribution ranking — a number near +1 means the leaderboard predicts reality, a number near 0 means it doesn't. They wrap it in a twelve-tier apparatus with three falsifiable criteria, so a benchmark's claim to validity is something you can test and reject, not just assert. In production terms, it is the difference between trusting an offline leaderboard and watching how rankings hold under shifted, online traffic.

How you read the benchmark	What it reports	What it misses
Aggregate score (today's leaderboard)	one mean number per agent → a sorted ranking	whether that ranking survives any change in conditions
Score + confidence interval	the mean plus its in-sample noise	still in-sample only — no view of the out-of-distribution shift
Predictive validity (IBM)	rank correlation between in-sample and out-of-distribution rankings	— (directly tests transfer; ~14 implementations, 12-tier apparatus, 3 falsifiable criteria)

Where the ranking breaks

Here is why an unstable ranking is worse than a noisy one. Take an illustrative slice of three agents — call them A, B, C — that an aggregate-score leaderboard ranks A > B > C by a hair: scores of 71, 70, 68. The gaps are tiny, but the leaderboard reports a confident order, and a team reading it ships A. Now shift the distribution — a new asset class, a different orchestration — and re-score: A drops to 64, B holds at 69, C climbs to 67. The out-of-distribution order is now B > C > A, the exact reverse of where A and C started. The rank correlation between the two orderings is negative — the leaderboard didn't just lose precision, it pointed at the wrong agent. (Only the 14 implementations, 12-tier apparatus, and 3 falsifiable criteria come from the paper; the A/B/C scores are illustrative.) A single aggregate number with a tidy sort hid the one fact that mattered: that order was never stable enough to ship on.

Goes deeper in: AI Agents → Evals & Diagnostics → Pass/Fail vs Score

Related explainers

This explainer stands alone from its news item (one concept), so the closest neighbors are other explainers on how a single evaluation number can quietly mislead:

WeaveBench — trajectory-aware vs outcome-only grading — the sibling failure: WeaveBench shows a single run's grade can be inflated; predictive validity shows a whole ranking can be invalid
FutureSim — harness-level agent eval — evaluating the agent's process rather than a single final number, the same "one score hides the truth" theme
Effective Feedback Compute (EFC) — another result that a headline number (raw compute) is the wrong predictor of agent success

Continue in trackProduction Evals: detect when a ranking drifts under shifted traffic

Where the ranking breaks

Related explainers

Frequently Asked Questions