ARE-bench — In-progress workspace replay — What does it mean?

The news. On November 10, 2026, a 27-author team posted ARE-bench (arXiv 2611.14364): 75 tasks cut from recruited researchers' own agent sessions and full project folders, spanning seven research stages (Survey, Design, Data & Env., Code, Execution, Analysis, Writing) and graded by 1,173 binary checklist items. Seven frontier models ran three rollouts per task. Averaged over the three rounds, the strongest, GPT-6.1-Sol, passed every item on 66.2% of tasks, Claude Opus 5.5 on 56.9%, and the other five stayed below 40%. Read the paper →

Picture testing a contractor two ways. In the showroom, you point at a display sink and say "fit a faucet"; the only way to fail is to fit it badly. On a real job, the same request arrives in a kitchen that is half done: the client already decided to keep the oak cabinets, a box of tiles from a dropped plan is still in the corner, and the sink was installed last week by someone else. The same short request now carries rules nobody repeated: do not rip out the cabinets, do not use the old tiles, keep the sink that is already there. A contractor who aces the showroom can still break every one of them.

Research agents face the same gap. A real request such as "fix the training script after the failed run" can depend on methodological choices made in an earlier design discussion, and it lands in a folder with superseded drafts, misleadingly named files and paused runs. In agent terms, the request only has a meaning together with the agent's state: the history and artifacts accumulated so far. Many research-agent benchmarks remove exactly that state by starting from a clean or curated repository with a standalone goal, so they never check whether the agent respects it.

ARE-bench keeps the mess on purpose. Each task is built from a recorded session in three moves:

1. Cut at a real user request. A cut is discarded if it falls inside a file write or if the replay disagrees with the transcript.
2. Replay the recorded file events up to that request, with no cleanup. A task workspace holds 41.1 files on average, and in 65.3% of tasks some of them were written by the agent in earlier turns.
3. Summarize the earlier requests that still bind into a Background section: on average, the Background compresses 36 earlier user turns into a state summary that cites 10.3 still-effective requests. Plans the earlier agent never executed are left out, because they would give away the answer.

The grading rubric is written from what the user wanted. Later feedback in the session can sharpen an item, but only if a competent researcher could have inferred it at the cut point, and the rubric stays hidden from the agent, so the instruction never turns into an answer key. In track terms, each task is a golden case whose input is a whole project at one moment, not a single prompt.

The paper's central measurement isolates what the replayed context adds. It keeps every agent delivery and every per-item verdict fixed and changes only which items count. Take Kimi-K3: 75 tasks × 3 rollouts = 225 runs. Counting only delivery contracts (the explicitly requested artifacts) and general professional norms, Kimi-K3 scores a three-round mean of 69.78%, which is 157 of those 225 runs. Then add the inherited and workspace items, the requirements that come from Background, the papers and the code on disk. 93 of those 157 runs miss at least one of them, leaving 64 / 225 = 28.44%. DeepSeek-V4.1-Flash and GLM-5.3-Flash start from the same ~69% and land at 40.00% and 31.56%, and the gap between DeepSeek and Kimi grows from 0.44 points to 11.56 points once the full checklist is applied.

Back in the kitchen, the typical failure is not a missing faucet; it is a faucet that runs but is plumbed into the wrong pipe. The paper's case study shows this. In a repair task for a point-cloud model called CylindFormer, two agent runs both got training running again, with a nonzero loss and parameter updates. One passed 20 of 20 checklist items; the other passed 16, because it had changed the coordinate handling, the overlap labels and the matching losses that the reference paper in the workspace required. A run that only checks "does training resume?" scores those two repairs the same. Across all seven models, correctness items were the weakest dimension, and 85.5–92.8% of failed runs contained at least one unmet correctness item.

The scoring rule matters as much as the starting state. With partial credit (CSR), the seven models span only 13.1 points (79.66% to 92.72%); with all-or-nothing ISR they span 40.4 points (25.78% to 66.22%). Of GPT-6.1-Sol's 76 failed runs, 47 missed only one or two items. Partial credit makes almost-usable deliveries look nearly done, while an all-items-pass rule separates the models, which is the pass/fail versus score choice applied to a research handoff: a repair that breaks the method is not 80% of a repair.

Two limits are worth keeping in view. The benchmark has 75 tasks from research projects only, so the exact numbers describe that distribution, and the verdicts come from an LLM judge rather than human graders; the paper reports a separate judge-agreement check, but a judge is still part of the measurement. The transferable idea is the construction, not the leaderboard: if your agent works inside long sessions, build some eval cases by cutting a recorded session at a request and replaying its state, then grade the rules that state implies.

Goes deeper in: AI Agents → Evals & Diagnostics → Golden Cases

Related explainers

• Taste-Bench: grading one decision fork with later work hidden — another way to start an eval from a real mid-task prefix
• SWE-DIVE: a simulated user that reacts to the agent's changes — adds the user's side of the session instead of its history
• OSWorld2.0: long-horizon failure modes — why grading the final state exposes failures a single step hides