SWE-DIVE is a framework that turns single-turn coding benchmarks such as SWE-bench into multi-turn evaluations. An LLM user simulator talks to the coding agent, a validator checks the simulator's messages, and the original benchmark tests decide success.

Why not just use SWE-bench?

SWE-bench gives the agent one complete, well-written issue. Real sessions are iterative, with vague requests, pushback and changed requirements, and a single pass rate cannot show wasted edits or broken tests along the way.

How reliable are the results so far?

They are early. The paper is a non-archival workshop paper with a qualitative case study on one SWE-bench Verified task and a small pilot, so it demonstrates the method more than it ranks agents.

What do the three personas test?

The Expert Nitpicker tests whether an agent can act on precise review feedback. The Vague Requester tests whether it makes unrequested changes when instructions are broad. The Mind Changer tests whether it can adapt when the goal changes.

Agent· 2026-10-26

SWE-DIVE coding-agent benchmark — Dynamic user simulation — What does it mean?

Jargon

SWE-bench: A benchmark that gives a coding agent a real GitHub issue and a repository, then runs tests to see whether the issue was fixed.
User simulator: An LLM that plays the user. It reads the agent's reply and the code changes, then writes the next message in the conversation.
Persona: A fixed behavior pattern for the simulator: Expert Nitpicker, Vague Requester, or Mind Changer. These are controlled test conditions, not real user groups.
Conversation validator: A second LLM that checks each simulated message against the task, the plan and the persona, and sends it back with a reason if it fails.
Harbor: The container framework that runs the agent, the simulator, the validator and the test environment together in Docker.
Regression: A test that passed before the agent changed the code and fails after. It shows damage that a single pass on the target bug can hide.

The news. On October 21, 2026, researchers from the University of Innsbruck and Microsoft posted SWE-DIVE, a framework that turns single-turn coding benchmarks into multi-turn conversations with a simulated user. It was accepted at a NeurIPS 2026 workshop (non-archival), and the evidence so far is a one-task case study plus a small pilot, so treat it as an early method, not a settled result. Read the paper →

Picture a restaurant that trains its waiters with a printed order card: one clear, complete card, handed over once. A waiter who aces every card is not yet tested on a real diner, who orders vaguely, sends food back, and sometimes switches from fish to steak halfway through the meal. The card is a benchmark issue. The diner is a user. SWE-DIVE is the version of the test that uses a diner.

A standard coding benchmark such as SWE-bench hands the agent one complete issue and then runs the tests. SWE-DIVE replaces the printed card with a simulated user that talks to the agent over several turns. The simulator is an LLM given the original task and its reference solution. Each turn it reads the agent's latest reply plus the git diff of what the agent changed, then writes the next message: a first subtask, a correction, an answer to a question, or a new requirement. When it judges the task done, it returns done; a maximum turn limit stops agents that cannot finish.

The simulator plays one of three roles, taken from observed real users in the SWE-Chat study. The Expert Nitpicker keeps the goal fixed and sends precise corrections. The Vague Requester gives broad instructions and leaves decisions to the agent. The Mind Changer reverses the goal midway. Same task, three diners, three different tests of the same agent.

A simulated user can drift out of character or give bad guidance, which would corrupt the measurement. So a second LLM, the conversation validator, checks every generated message before the agent sees it. It must stay aligned with the original issue, follow the planned subtask path, match the assigned role, and be actionable. If a message is invalid, the validator returns a reason and the simulator regenerates it. The agent is graded only on messages the validator approved. The original benchmark's tests then decide whether the task is solved, as in the golden-case idea of a fixed, checkable answer.

Think of the evaluation as two questions. Offline, a frozen benchmark asks whether the agent can solve a well-written ticket. SWE-DIVE asks a different one: whether the agent still behaves when the ticket is badly written and the user keeps talking. That second question is closer to what you would otherwise only learn from online traffic, and it fits the pass/fail-versus-score lesson: a single bit of pass or fail cannot show waste or regressions.

Worked example (illustrative numbers). Suppose the reference run fixes a bug by editing 1 file and spending 20,000 tokens, and the project has 180 tests that passed before the change. A Vague Requester run also fixes the bug, but edits 6 files and spends 52,000 tokens, and 10 of the 180 earlier tests now fail. The paper defines waste as edits to files the reference run left alone plus extra tokens, and regression as a test that passed before and fails now. Here waste is 5 extra files and 32,000 extra tokens (+160%), and regression is 10 of 180 tests (5.6%). A plain pass/fail on the bug alone would have hidden both. The paper's real case study, on one SWE-bench Verified task, showed the same pattern: the Vague Requester run fixed the bug but broke 10 previously passing tests, while the baseline run made a one-file, one-line fix.

The framework has a stated limit: it uses the git diff as its progress signal, so a benchmark must expose one, or a different signal must replace it. The paper also hedges its own design. Splitting a task into subtasks could make the benchmark easier, which is why the authors plan to test smaller models, where the effect is easier to see.

Goes deeper in: AI Agents → Evals & Diagnostics → Golden Cases

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based