Agent·

When Agent Skills Go Stale — Selective skill maintenance by replaying execution checks — What does it mean?

The news. On November 17, 2026, researchers from AWS Agentic AI and the University of Southern California posted When Agent Skills Go Stale: Runtime Behavior and Selective Maintenance in Evolving Environments (arXiv 2611.22525). They reused the same saved skills before and after real environment changes on four benchmarks — TerminalBench-2, historical IRS tax forms, AppWorld and ScienceWorld — with Claude Haiku 4.5 and Claude Opus 4.7. Previously helpful skills could lower task performance by more than 10 percentage points or raise execution cost by nearly 40% against using no skill, and their maintenance method recovered up to 67 points of accuracy. Read the paper →

Picture a courier who keeps a route sheet. Every line on it was true the day it was written: turn right on Main St, left on Elm St, unload at Dock 4. Then the city renames Elm St to Oak Ave. The sheet is now wrong in exactly one line and right in every other, and nothing about the sheet tells the courier which line that is. A careful courier notices the new street sign and turns anyway; a hurried one follows the sheet and gets lost. Either way the sheet itself stays wrong, so the next shift pays the same price again.

An agent skill is that route sheet. The paper calls what it carries skill state — instructions, code and values the agent saved in the original environment W0 so it does not have to rediscover them. A concrete case from the paper: a tax-form skill writes the taxpayer's name into the PDF field f1_1[0]. In the 2024 revision of Form W-9 that field is called f1_01[0]. The write does not error; it is silently ignored, so the form comes back with an empty name line. This is the agent-side version of upstream drift you do not control: nothing in your own code changed, yet yesterday's behaviour no longer describes today's.

Why a stale skill can be worse than no skill

The paper measures a skill's value against the no-skill baseline — the same agent doing the same task without the skill — on two axes: task score and execution cost (agent turns, or environment steps on ScienceWorld). After the change, the two axes move separately. On ScienceWorld, Opus 4.7 with the stale skill scored lower and needed 38.9% more environment steps than with no skill at all. On the tax forms it still saved turns but filled fields less accurately than the baseline. A skill can look cheap and be wrong, or look correct and quietly stop saving work, so task success alone cannot tell you whether a skill still pays.

How a skill was written decides how badly it breaks. The authors wrote two versions of each skill from the same material. A cache-heavy skill stores the answer (“the field is f1_1[0]”); a procedural skill stores how to find it (“list the form's fields to confirm the names”). Back on the route sheet, that is the difference between writing down the door code and writing “ask the front desk for the code”. On TerminalBench-2 and the tax forms, cache-heavy skills lost more of their benefit after the change in all four model-and-benchmark comparisons, and saved more work before it in three of the four. On the revised IRS forms, Opus 4.7 with the procedural skill kept a 6.2-point gain over no skill, while the cache-heavy skill dropped it 12.5 points below the baseline. The pattern breaks on AppWorld and ScienceWorld, where both versions stored the same outdated endpoint or room, so lookup steps had nothing to recover.

How selective maintenance works

The core idea is to record, while the skill still works, what the environment looked like to it — and later to replay those checks as plain code before every reuse. In the courier's terms, each line of the route sheet gets a tick box that says what was observed the day the route worked: the sign read “Elm St”.

  1. Record checks. Rule-based extraction pulls named paths, commands and identifiers out of the skill's text and code, and the system re-runs operations from successful runs — a form write with a placeholder value, an API call. Each check is stored with its W0 output and a link back to the line of the skill that depends on it. Checks whose output varies between repeated W0 runs are thrown away, so a flaky check cannot raise a false alarm later. This is the same discipline as a golden case: a fixed input with a known-good result.
  2. Replay before reuse. The checks run in the current environment without any model call; measured median replay times were 0.2 to 4.3 seconds on the tax-form, AppWorld and ScienceWorld setups. If every output matches, the skill is reused unchanged.
  3. Verify. A changed observation is not automatically a stale skill, so each difference is checked against what the skill actually relies on — directly from the profile for simple cases, by an LLM verifier when interpretation is needed. In the TerminalBench-2 control runs, where the change did not affect the skill, verification rejected most of the replay flags.
  4. Repair, else rebuild. An LLM maintainer edits only the conflicting part of the skill and keeps the rest. If the conflict cannot be pinned to one place, or the repair keeps failing validation, the skill is rebuilt from scratch in the new environment, without looking at the stale version.
  5. Validate. The candidate runs on held-out validation tasks and must meet thresholds on both score and cost before it replaces the old skill — the same gate an A/B harness applies to a prompt change. If no candidate passes, the existing skill stays.

Repair is tried first for a cost reason as well as a quality one: on the tax forms, rebuilding used 1.24 to 2.28 times as many model tokens as repairing. Neither path consistently produced the more accurate skill, which is why the paper keeps both.

Benchmark (what changed)Task score, stale → maintainedCost, stale → maintainedSource
TerminalBench-2 (packages, config, data schema)0.561 → 0.64938.9 → 32.1 turnsTable 4
IRS forms (fields renamed, renumbered, moved)0.305 → 0.9757.0 → 4.7 turnsTable 4
AppWorld (three API endpoints renamed)0.627 → 0.66728.7 → 23.3 turnsTable 4
ScienceWorld (an object moved rooms)1.000 → 1.00028.0 → 18.6 stepsTable 4

Haiku 4.5 with the cache-heavy skill in every row. Opus 4.7 results are in the paper's appendix. The procedural AppWorld skill is the one setting where mean task score went down after maintenance (0.627 → 0.569).

Worked example: is maintenance worth its tokens?

Take the tax-form setup with Haiku 4.5 and the cache-heavy skill, and hold three numbers from the paper's Table 5 fixed. One maintenance event — verification, writing the candidate and validating it — costs 142.8K model tokens, once. After that, each tax-form task costs 14.58K tokens with no skill and 7.85K tokens with the maintained skill, a saving of 6.74K tokens per use. Divide the one-time cost by the per-use saving: 142.8K ÷ 6.74K ≈ 21 uses to break even against simply retiring the skill. The procedural Haiku skill saves less per use (3.60K), so it needs about 42 uses. So maintenance is an amortization decision: worth it for a skill the agent will reuse dozens of times, not for one it touches twice. Against keeping the stale skill, the comparison is different again — the stale skill already finished in 7.0 turns but filled only 30.5% of fields correctly, versus 97.5% in 4.7 turns after maintenance, so there the payoff is mostly accuracy. The paper notes this estimate assumes future tasks cost the measured average and need no further maintenance in between.

Where the method stops

Back on the route sheet: a tick box only covers what was written next to it. Replay can only catch changes to the properties its checks recorded — the paper reports misses when a file's contents changed in a way no check observed, and some control runs still triggered repairs that were not needed. The experiments also cover a single maintenance event; refreshing the check profile after an accepted update is left to future work. The practical lesson carries further than this paper: if a skill stores a resolved value, pin it to the version it was resolved against and give it a check, because a silent no-op write is the failure your evals are least likely to see.

Goes deeper in: AI Agents → Tool Use & Function Calling → Skills as a Primitive

Related explainers

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based