What is a decision fork in Taste-Bench?

A point in a recorded agent run where two directions were both reasonable and the later work showed which was better. Taste-Bench freezes the run there, hides everything after it, and asks a model to choose; the recorded outcome of each branch provides the label.

Why doesn't more reasoning improve an agent's taste?

On the hardest forks the deciding evidence only appears in later work, so it is not in the prompt to reason about. Raising reasoning effort on two GPT-5.6 models changed accuracy by at most **2.2 points**, and the models spent the most tokens where they were least accurate.

How does Taste-Bench relate to end-to-end benchmarks like SWE-bench?

End-to-end benchmarks score whether a task finished; Taste-Bench scores the choices made along the way. The two are only partly correlated (r = +0.63 with SWE-bench Verified), and models tied at the top of SWE-bench Verified are 10.7 points apart on Taste-Bench.

Can taste be trained?

Yes, in the paper's setup. A teacher shown the winning option writes its reasoning, and a student without that answer is trained to match it. The student rose from 30.0% to 47.9% on unseen tasks, and its advice lifted an executor's SWE-bench Pro success from 14.6% to 33.7%.

Agent· 2026-09-24

Taste-Bench: best model picks the right fork 59.7% of the time — Outcome-hidden decision forks — What does it mean?

TL;DR

What is it: Taste-Bench (arXiv 2609.25804) is a benchmark of 502 decision forks mined from real agent runs: at each fork a model sees the run so far and two possible directions, and must pick the better one before any later work is visible.
Why it’s needed: A long agent run is decided by a few directional choices, such as which hypothesis to test or which fix to build on, and a pass rate alone cannot tell a good choice from a lucky one, so a direct score for those choices gives evaluation and training a new target.
vs previous: End-to-end benchmarks such as SWE-bench score only whether the whole task finished, which mixes decision quality with execution and luck; Taste-Bench isolates the decision by comparing branches that start from an equivalent prefix, and its ranking correlates only partly (r = +0.63) with SWE-bench Verified.

Jargon

Decision fork: A point in an agent's run where two directions are both reasonable and the later work decides which was better. Taste-Bench turns each one into a two-choice question.
Trajectory prefix: Everything the agent saw and did up to the fork: observations, tool calls and outputs. The evaluated model reads this and nothing after it.
Parallel vs detour fork: A parallel fork comes from two separate attempts at the same task that split and end differently. A detour fork comes from one run where the agent took a direction, saw it fail, and switched.
Time horizon (of a fork): How far past the fork you must look before the better option is clearly justified, from in prefix (already visible) to more work (needs a later check or substantial work).
Position bias: The tendency of an LLM to prefer the first (or second) option regardless of content. Taste-Bench asks each question in both orders and counts it only when both answers are right.
Context distillation: Training a model to produce, without some extra context, what the same model produces with it. Here the extra context is which option won. See also DOPD on the risk of privileged teachers.
SWE-bench Pro: A benchmark of long software-engineering tasks on real repositories, graded by running tests. Taste-Bench's engineering forks come from 2,677 graded rollouts on 517 of its tasks.

The news. On September 22, 2026, researchers from City University of Hong Kong and Microsoft, with one independent co-author, posted the paper "The Tasteful Agent" (arXiv 2609.25804) and released Taste-Bench. It holds 502 two-choice questions: 390 from software-engineering runs on SWE-bench Pro and 112 from AI-research runs in METR's public MALT transcripts. The best of 14 frontier models scores 59.7%. A Qwen3.6-27B student trained from an outcome-aware teacher rises from 30.0% to 47.9% on questions from tasks it never saw. Read the paper →

Picture the hiker at the junction. Both trails look fine for the first few metres; one reaches the summit, the other ends at a cliff an hour later. A decision fork is that junction inside an agent's run: a point where two directions are both plausible and only the later work shows which one was right. For a coding agent the two trails might be "patch the caller" versus "fix the shared helper"; for a research agent, "tune the learning rate" versus "change the data mix". An end-to-end benchmark only asks whether the hiker reached the summit. It cannot tell a good choice at the junction from a bad choice followed by luck, because execution quality and the environment also decide the outcome.

Taste-Bench gets its answer key from hikers who already walked both trails. Agent systems often run the same task several times, and when two attempts share an equivalent prefix and then split, the recorded outcomes of the two branches label the better direction, with no human annotation. That is the parallel construction. The detour construction uses one run in which the agent took a direction, saw it fail, and switched; the abandoned direction and the recovery become the two options, and the fork is placed just before the mistake. Then the fog comes down: the benchmark freezes the run at the fork, hides every later step, and asks the model to choose.

To keep the questions honest, judge models drop any fork that can be answered from the wording of the two options alone, and any fork whose label the full record does not clearly support. Only 10.8% of 4,657 proposed forks survive. In a human check of 100 sampled questions, reviewers who saw the recorded outcomes agreed with the mined label on 170 of 172 explicit judgments (98.8%). Each question is asked twice with the options swapped and counts only if both answers are right, so random guessing scores 25% (½ × ½) and always picking the first option scores 0%.

The results say taste is weak, and that thinking longer at the junction does not fix it. Each fork is tagged with a time horizon, shown in the table below. Averaged over the 14 models, accuracy falls from 62.3% when the deciding fact is already in the prefix to 21.0% when it needs more later work, below the 25% random baseline. Raising reasoning effort on two GPT-5.6 models changed accuracy by −0.2 and +2.2 points, and both models spent the most reasoning tokens at the horizon level where their accuracy was lowest. That is the hiker again: staring harder at the fog does not show you the cliff. For an agent builder it is a warning against treating a bigger reasoning budget as the fix for bad direction choices.

Time horizon	Where the deciding evidence is	Mean accuracy, 14 models
In prefix	A fact already visible before the fork rules one option out	62.3% (paper, Fig. 4)
Inferable	No single decisive fact, but hints in the prefix together justify the answer	not given in the text; plotted in Fig. 4
Next step	The first observation after the fork settles it	not given in the text; plotted in Fig. 4
More work	Only a completed check or substantial later work settles it	21.0%, below the 25% random baseline (paper, Fig. 4)

Taste is a separate score, not a restatement of end-to-end skill. Against public SWE-bench Verified scores across 11 models, the correlation is r = +0.63, and only +0.37 on the engineering subset, even though that subset was mined from SWE-bench Pro. The four models at the top of SWE-bench Verified are within 4.0 points of each other there but 10.7 points apart on Taste-Bench. This is the lesson of moving from a single pass/fail bit to a scored rubric: the final pass rate hides where a run went wrong, and a per-decision score shows it.

The paper also shows that taste can be trained. A teacher that is told which option won writes out its reasoning, and a student that sees only the task, the prefix and the two options is trained to reproduce those reasoning tokens. Teacher and student are the same Qwen3.6-27B base model with different contexts, and training updates only LoRA adapters (small low-rank add-on weights), so the judgment that the answer made possible moves into the student's weights. The authors chose this over fitting the bare A/B labels, which carry one bit per question and, they argue, let a model memorize answers without learning the judgment. On questions from tasks it never trained on, the student goes from 30.0% to 47.9%.

Here is what that judgment is worth end to end. Hold the executor fixed (a Qwen3.6-27B agent) and the task set fixed (41 held-out SWE-bench Pro tasks), and change only the fork-derived advice placed in its context before the run starts. The forks were mined from earlier runs on the same tasks, so they are known before the new run starts. With no advice the executor solves 14.6%, about 6 tasks. With correct advice at every fork it solves 39.0%, 16 tasks, the upper bound for this kind of advice. With the student's advice it solves 33.7%, about 14 tasks. The student's advice captures 19.1 of the 24.4 points that perfect advice adds, about 78%, without changing the executor model.

Goes deeper in: AI Agents → Evals & Diagnostics → Pass/Fail vs Score

Related explainers

WeaveBench: trajectory-aware vs outcome-only grading — also looks inside the run instead of at the final state, but to catch shortcuts, not to score choices
Critical-State RL: nested continuation sampling — separates what one action controls from later-turn noise, for training rather than evaluation
DOPD: dual on-policy distillation — when a teacher's privileged context helps a student and when it misleads

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based