AfterDeleteBench — deleted memory, surviving summary — What does it mean?
The news. On October 22, 2026, researchers at The Ohio State University released AfterDeleteBench, a behavioral audit for persistent-memory LLM agents. It deletes a synthetic secret's source record, then checks whether the agent can still produce the secret. Qwen2.5-7B-Instruct recovered it in roughly two thirds of primary trials, and never-exposed controls recovered none. Read the paper →
Picture an assistant who types up the meeting minutes, and only afterwards is told to shred the original letter. Shredding the letter does nothing to the minutes, and the minutes still say the code. A persistent-memory agent works the same way: it stores each message as a source record, and later builds derived state, such as a summary, from those records. A delete button that targets the record removes the letter and leaves the minutes on the desk.
The agent in this benchmark does not read the minutes by looking them up. The whole post-deletion memory is placed in its prompt, so the surviving summary is simply part of what the model is told before it answers. Any later question that touches the secret can be answered from the minutes, with no trace that a letter ever existed.
To measure this without guessing, the authors run paired worlds for each synthetic secret. In the forget world the agent is shown the secret, memory is consolidated into a summary, and then the source record is deleted and verified gone. In the clean world a neutral message replaces the secret and the same steps run. Model, memory design, auditor, query budget and scoring rule are shared. Because the clean agent never saw the secret, a correct guess in the forget world is attributed to the earlier exposure, because the clean agents recovered none. This is a test of what the agent's state object keeps, not of whether a database row was removed.
The scored event is strict: the auditor's one submitted answer must equal the secret exactly. Memory design matters. With flat memory, which has only raw records, deleting the source removed the intended copy and no recovery was observed. With hierarchical memory the summary survives exact deletion, and that is where recovery appears.
Here is the numbers walk-through, holding three things fixed: the model (Qwen2.5-7B-Instruct), one direct query, and hierarchical memory with no write filter. The paper reports a recovery gap of 0.680 on in-distribution profiles with clean recovery at 0 of 200. Since the clean world recovers none, the gap equals the forget-world rate, so about 0.680 × 200 = 136 of 200 secrets came back (our arithmetic from the reported rate, not a table count), against 0 of 200 for never-exposed agents. On held-out profiles the gap is 0.640, about 128 of 200. So roughly two of every three deleted secrets were still one question away, and the control shows the agent was not simply guessing.
A natural defense is to search memory for the deleted text, so the authors also removed the complete secret. In an illustrative version of their component test, a secret shaped like REGION-MNEMONIC-NUMBER is stored as three separate facts plus a public rule for combining them (the paper uses that form). The complete string appears in 0 of 1,248 model-visible states, yet exact recovery gaps range from 0.615 to 0.990 across six model and split cells. A scan for the literal secret finds nothing while the agent can still rebuild it from the parts. Provenance-aware closure removed this channel in every tested cell, because it retired the derived state that held the parts.
| Control | What it acts on | Result on reconstruction | Result on write-back |
|---|---|---|---|
| Exact source deletion, hierarchical memory | The addressed record only | Gap 0.680 in-distribution, 0.640 held-out (controlled synthetic secrets) | Not the focus; the summary still answers |
| Exact source deletion, flat memory | The addressed record only | Zero recovery observed in the tested cells (controlled synthetic secrets) | Not applicable |
| Recorded-lineage closure | The source and every summary built from it | Zero observed gap under complete provenance (idealized upper bound) | Zero observed gap |
| Exact-literal write gate | A proposed future write | Unchanged; earlier residue stays available | Blocks the re-save entirely |
The last two rows show that the controls work at different stages of the memory lifecycle. Closure acts on state that is already stored. The gate acts only on a write that has not happened. Using only the gate is like banning anyone from retyping the letter while the minutes stay in the folder. For an agent that holds private data and talks to outsiders, this separation decides which promise you can make: the gate stops the secret from coming back, but only closure removes what is already there. The same drift is one of the context failure modes: stale or sensitive content that stays in the window after it should have left.
There are limits, and the paper states them. The secrets are synthetic exact strings, the summary is a deterministic word-for-word join, and closure works here because provenance is complete, so it is an upper bound and says nothing about learned, lossy or untracked memory. Zero full-secret recovery also does not mean zero leakage: when the number part was omitted, the region and mnemonic parts still appeared in the replies in most forget-world profiles (for example 93 of 104 for Qwen in-distribution). A deletion audit should score partial leakage, not only the full secret. Finally, the way the reply is parsed changes the answer, so an audit with a mismatched response format can report zero for replies that actually contain the secret.
Goes deeper in: AI Agents → The Agent Loop & State → The State Object
Related explainers
- MemOps: memory lifecycle operations — grades remember, forget, update and reflect as separate operations
- MHG-Mem: provenance hypergraph memory — the lineage records that make closure-style deletion possible
- GateMem governance trilemma — the trade-offs of controlling what memory may keep