The news. On July 9, 2026, the authors of Long-Horizon-Terminal-Bench published a benchmark of 46 long-horizon terminal tasks across nine categories — including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, and is decomposed into fine-grained graded subtasks so agents earn dense intermediate rewards and partial credit. Evaluating 15 frontier models, the paper reports runs averaging 9.9M tokens, 231 episodes, and 85.3 minutes per task, with the strongest result reaching 15.2% pass@1 at a 0.95 partial-reward threshold and 10.9% at a 1.0 perfect-reward threshold. In the paper's words, these tasks stress "long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving." Read the paper →
Picture a marathon where the only instrument is a camera pointed at the finish line. One runner pulls up injured at the first kilometre. Another arrives at the tape, sees it, and collapses a stride short. Develop the photo and both runners produce the same result: no finisher. A single pass/fail verdict compresses 9.9 million tokens, 231 episodes, and 85.3 minutes of running into one bit — and that bit cannot tell a total collapse from a near miss. For a one-shot problem that is a fair trade, because there was barely a course to run. For a task that takes hundreds of episodes, it is most of the information you paid for, thrown away at the moment of scoring.
So the benchmark lays timing mats along the course. Each of the 46 tasks is cut into fine-grained graded subtasks, and the agent earns dense intermediate reward as it clears them, so a run's score records how far it got rather than only whether it arrived. This is what "grades progress densely" means, and it is why the setup matters: each task follows a Terminal-Bench-style setup with a reference solution or a simulation engine, which is how this benchmark makes a subtask checkable at all. Mats are cheap to read but expensive to lay — someone has to decide, per task, what counts as reaching kilometre 30. That work is the benchmark.
Now the part that is easy to miss. Once a run produces a fraction of reward rather than a verdict, something has to turn that fraction back into a pass or a fail for the leaderboard — and that something is a number a human picks. The threshold is not a property of the agent; it is a dial the evaluator chooses, and it lands in the headline as if it were a measurement. Set it at 1.0 and you are back to demanding a perfect run. Set it at 0.95 and a run that cleared 95% of the graded reward now counts as a finish.
| Where the line is drawn | What a run must achieve | Best reported pass@1 |
|---|---|---|
| 1.0 (perfect reward) | the task's full graded reward — no credit for a near miss | 10.9% |
| 0.95 (partial reward) | 95% of the task's graded reward | 15.2% |
| no threshold | report the reward earned and skip the pass line entirely | not a rate — the progress signal the mats record |
Put the numbers on the course. The benchmark holds 46 tasks fixed and gives each model one attempt each. At the 1.0 perfect-reward threshold, the best of the 15 frontier models clears 10.9% pass@1 — on 46 tasks that is about 5 tasks (arithmetic on the paper's reported counts). At a 0.95 partial-reward threshold, the best reported result is 15.2%, or about 7 of 46. A threshold is applied when an already-graded run is scored, so what separates those two rows is where the pass line sits rather than anything an agent did differently. (The paper reports the best result at each line, which need not be the same model both times.) Two tasks' worth of headline sits inside a five-point choice about where to draw the line — roughly 39% relative, from 10.9% to 15.2%. That is the whole lesson of the threshold in one comparison, and it is why the number is worth naming out loud instead of leaving in a footnote: a reader who sees only "15.2%" cannot reconstruct which line produced it.
None of this makes an agent better at a long task, and the paper is not claiming it does. Both numbers are brutal — at the strict 1.0 line, nearly 9 of every 10 of these 46 tasks come back a fail even for the best of the 15 models, which is what one benchmark's tasks say about long-horizon terminal work today. What the mats buy is diagnosis: a run that dies at kilometre 2 and a run that dies at kilometre 41 are different failures with different fixes, and grading only the end state cannot tell them apart in your metrics. The threshold is where a graded course gets flattened back into a rate, so it is worth asking about before you trust a long-horizon agent score — including your own. When you next read a result like this, the useful question is not "how high" but what had to be true for that to count as a pass.
Goes deeper in: AI Agents → Evals & Diagnostics → Pass/Fail vs Score
Related explainers
- WeaveBench — Trajectory-aware vs outcome-only grading — the sibling question: not where the pass line sits, but who reads the run to place it
- OSWorld2.0 — Long-horizon computer-use failure modes — what the near-misses actually look like once you can see them
- Agentic CLEAR — System/trace/node eval granularity — zooming a diagnosis in and out across the same trace
- Plan evaluator study — Deletion non-monotonicity — the other way a score misleads: when the cheapest way to raise it is to do less work