Chronicle replays agent incidents as CI tests — Cut-point replay vs stubbing every boundary — What does it mean?
The news. On September 17, 2026, a paper called Chronicle (arXiv 2609.20625) proposed a record-and-replay system for LLM agents built around one operation: cut-point replay. Chronicle records an agent run at its nondeterministic boundaries as immutable envelopes, then replays a chosen subset from that record while the complementary subset executes live against changed code. On a benchmark of six recorded failures with simulated model boundaries, recording adds 23 microseconds per boundary crossing, full replay issues zero model calls and is bit-stable across 20 repetitions, and the cut-point tests fail on faulty code while passing once a guard is added, and on benign changes. Read the paper →
Picture the karaoke night. The band is on tape, the singer is live, and that is the only arrangement in which you can tell whether the singer is any good. Play the full studio recording instead and you learn nothing about the singer, because nothing they do reaches the speakers. Send the band home and no two nights sound alike, so you cannot separate a bad night from a bad singer. The useful setup is the mixed one: enough on tape to make the night repeat, and exactly the part you want to judge left live.
That mixed setup is cut-point replay. An agent run touches the outside world at points where the answer is not guaranteed to repeat — a model call whose sampling is not bitwise reproducible, a tool that reads state which has since moved, a clock, a seed. Chronicle calls these nondeterministic boundaries and records each crossing as an immutable envelope — enough of the exchange to hand the same answer back later without making the real call. That much is what trace replay already gives you, and replaying the whole set reproduces the original run exactly — the paper reports zero model calls during full replay and bit-stability across 20 repetitions. But a run that replays everything is a recording, not a test: every boundary it serves from tape is a piece of code that does not run.
The cut point is where you choose. Serve the model boundaries from the record and the model's contribution to the incident is frozen at exactly what it was that night. Leave the tool boundaries live and your new guard code actually runs, against the same inputs that produced the original failure. The test then fails while the fault is still there and passes once your fix holds — which is what replaying a trace in anger is supposed to buy you, except it now runs unattended in CI instead of at 3am on one engineer's laptop.
This is also why the obvious shortcut does not work. Stub every boundary — hand each call a canned answer and assert on the outcome — and every call you stubbed is a call your code no longer makes. Chronicle's mutation study makes that concrete: with the same assertion, in a mutation study of the guarded tools, cut-point tests caught every mutant that let the recorded unsafe action through, while the stub-everything baseline caught none. The difference is not the assertion, and it is not the recording. It is which side of the cut your code sits on.
| Replay strategy | Served from the record | Model calls | Catches a mutant that lets the unsafe action through |
|---|---|---|---|
| Re-run the incident live | nothing | all of them | not measured — a multi-step run rarely repeats at all (paper) |
| Full replay | every boundary | zero, bit-stable over 20 repeats (paper) | no — a replayed boundary is code that does not run |
| Stub every boundary | every boundary, with canned answers | none, by construction | none (paper) |
| Cut-point replay | a chosen subset | only the ones left live | every one, in the mutation study of the guarded tools (paper) |
Work the overhead, because recording everything sounds expensive until you price it. Hold three things fixed: a model call takes 300 ms (the paper's assumed figure), recording one boundary crossing costs 23 microseconds, and the run crosses 40 model-call boundaries (illustrative — the paper reports a per-crossing cost, not a run length). Recording those crossings adds 40 × 23 = 920 microseconds, a bit under a millisecond. The same 40 calls cost 40 × 300 ms = 12 seconds of waiting on the model. So the record costs about 0.9 ms on a 12-second run, or 0.008% — and since both sides scale with the count, that share holds however many model calls the run makes. Add the tool and clock boundaries back in and the recording cost rises while the model wait does not, so the honest claim is the paper's own: 23 microseconds against a 300 ms call, whatever the mix. You are not trading recording against speed; you are trading it against having no way to reproduce the incident at all.
What it buys downstream is a supply of real test cases. A recorded incident that replays deterministically is a golden case you did not have to invent, drawn from a failure that actually happened, and a suite of them is the gate an eval-driven rollout needs before it promotes a prompt or a tool change. The catch is the same one every record-and-replay system has: the record is only as good as the boundaries you thought to instrument, and a failure that crosses a boundary nobody wrapped will not be in the tape at all.
Goes deeper in: Agent Engineering → Incident Handling → Trace Replay in Anger
Related explainers
- A serving study traces agent irreproducibility to prefix-cache state — why the same request diverges in the first place, which is the problem Chronicle records around.
- PreAct compiles agent runs into replayable programs — the same recording used to cut cost rather than to test.
- NVIDIA Blackwell leads AgentPerf, the first agentic-AI infra benchmark — a third use of a recorded trajectory: measuring the serving system under it.