Agent skill memory that rewrites itself from user traffic — Matched replay gate — What does it mean?

The news. On 18 September 2026, researchers from Adobe and Brown University published a continual-adaptation framework for a graphic-design agent that drives professional design software through more than 230 tools. Over five rounds on 1,406 briefs — real user traffic plus LLM-generated variants of it — the agent's skill bank grew from 76 documentation-derived skills to 139 — with no weight updates and no human reward labels. Every one of those changes was proposed automatically, and every one had to clear the same acceptance test first. Read the paper →

Picture the binder that sits on the pass in a busy kitchen. Each card is one procedure, written in plain language, pulled out when an order needs it — the agent's equivalent of deciding which tools and instructions to load into the context for the task in front of it. The binder grows along two axes, and neither one is allowed to write directly into it. Widening mints a card for an order the kitchen keeps improvising: an uncovered subtask is filed under a canonical label, and only once that label has recurred three times is a candidate card distilled from those cases. Deepening rewrites a card that keeps sending back bad plates: a run scoring below 0.6 on the system's own rubric counts as a failure against every skill it retrieved, and any skill carrying two such failures is queued for revision, worst first.

The rewrite is deliberately not written from complaints alone. The reviser is handed the brief, the per-requirement verdicts, the "why-bad" rationales, the current card — and a contrasting set of that same card's successful calls on similar tasks, down to the tool sequences that actually worked. It edits against a visible success-versus-failure divergence rather than guessing from failure text, and when repeated targeted edits to one card keep getting refused, it escalates to rewriting the whole card.

Now the part that makes this more than a self-editing text file. The naive acceptance rule — re-run the task, and ship the new card if the score went up — is confounded twice over. A vision-language grader's absolute score for the very same image moves between runs, and the agent's own execution is stochastic, so "the mean rose" can mean judge drift and solver noise happened to line up rather than that anything improved. On top of that, two runs of the same brief can retrieve different assets and start from different upstream state, which lets a candidate win on inputs rather than on merit.

The gate removes both by construction. The gate generates several contexts per replayed prompt, then freezes each one and replays it fresh in the same batch under both arms, so whatever drift that batch carries shifts the two arms equally and cancels. The grader is never asked for a score — only which of two outputs better satisfies the brief, with the presentation order swapped to blunt position bias. Those votes aggregate into a per-case win rate, a tie band around 0.5 absorbs jitter, and a prompt counts as won only if the candidate wins a majority of its own contexts, so one lopsided context cannot mask losses elsewhere. The ship rule is then blunt: no case may regress, and at least one must improve. A candidate that merely ties everywhere is refused. This is the same shape as the A/B harness you would put in front of a prompt change in production, with the replay set playing the role of your golden cases — except here the thing under test is a file the agent wrote about itself.

Hold three things fixed — five evolution rounds, the same 1,406 briefs (real traffic plus LLM-generated variants of it), and a model whose weights never move — and the gate's cost comes out in counted decisions. Deepening proposed 231 rewrites; the gate refused 100 and committed 131. Widening proposed 136 mints; the gate refused 67 and committed 69. Add those up: of 367 automatic proposals, 200 shipped, which means 167 proposals — about 46% — never reached the bank the agent actually reads from. The bank still ended the run at 139 skills, up from 76, a net gain of 63 rather than 69 because deepening also retired 6 skills it had superseded or merged. None of those verdicts is free, either: a rewrite is replayed on 4 briefs under 2 arms, so a single rewrite proposal costs at least 8 fresh agent runs — more once you count the several contexts per prompt, plus the pairwise judging on top — before a single word of the card changes. A mint draws its replay prompts from the triggering cluster instead, and compares the candidate against the agent with no skill retrieved.

What did the surviving edits buy? On Claude Sonnet 4, the round-5 bank raised GenEval2 success from 72.7% to 99.3% and generation quality by 11.99 points — and that pair of numbers belongs to one backbone on one benchmark, not to the method in general. The same table shows GenEval2 quality falling 3.32 points on Claude Opus 4.6 (while its success rate rose 1.3), and DPG-Bench losing 3.20 quality and 15.3 points of success on Qwen3.6-27B. Average quality still rose for all three backbones — though quality is only measured on runs that succeeded, so it reflects the easier prompts whenever the success rate is low — and mean latency per successful generation rose by only 3.4–6.2%.

The sharper lesson is what a passing gate does not promise. On the paper's internal benchmark — 200 fixed, human-authored briefs run with Claude Sonnet 4 — round 4 fell below the no-skill baseline at completeness ≥ 0.3, where completeness is the share of the brief's own requirements the output actually delivers: 90% against 94%, even though every change inside it had cleared the gate, because a freshly minted skill is only ever checked against the cluster that triggered it and can misfire on neighbouring briefs. Round 5, which committed 21 rewrites against only 4 mints, recovered that lower end and came out strongest at every threshold. The authors state the limit plainly: refusing regressions you can see on the replay set is not the same as improving over the whole traffic distribution. If you build one of these, the gate is your merge check, not your monitoring — you still want drift detection and shadow-mode coverage watching what the replay set never sees.

Goes deeper in: Agent Engineering → Production Evals → A/B Harness

Related explainers

• Self-evolving agents collapse over iterations — Continual experience internalization — the other route for the same instinct: folding past runs back into the weights rather than into an external file.
• SkillCoach self-evolves rubrics to grade agentic skill-use at scale — the judging side of the loop, where the grading criteria themselves are the thing being evolved.
• A 176-setting study ablates the coding-agent harness — what else around an agent is worth changing once the memory is under control.