Cut agent attacks 3× by co-evolving harness and policy — Harness-policy co-evolution — What does it mean?
The news. On September 2, 2026, a paper called SafeEvolve (arXiv 2609.02786) proposed treating an agent's safety harness and its model policy as one system that evolves together. Working with Qwen3.5-4B on AgentDojo — a benchmark that hides malicious instructions inside tool output — the authors take the agent's own completed runs, convert the safety failures in them into bounded edits to the safety prompt and to the hierarchical skills the agent follows, and then run a two-stage training recipe that teaches the model to use those edits. They report a threefold reduction in attack success rate, with benign utility rising from 59.79% to 61.86%. Read the paper →
Picture the kitchen. A near-miss ticket comes back from the pass: someone had slipped an extra line onto an order, and the line cook simply cooked it. There are two obvious responses, and most teams pick exactly one of them. You can pin a new rule card on the wall — check every ticket against the printed menu before you start — or you can pull the cook off shift for retraining. Pin the card and skip the drill and the cook walks straight past it; run the drill and skip the card and the cook has nothing concrete to check against.
That same either-or runs through agent security today. The harness is the kitchen wall: the wrapper code that formats the prompt, executes tools, filters what tools return, and decides what the model may call next. It is where engineers put their defense-in-depth layers, and it is what gets patched the morning after an injection lands, because changing it is a code or config change rather than a training run. The policy is the cook: the model weights that decide, turn by turn, what to do with the ticket in front of them. Retraining the policy is the other lever, and it is slow, expensive, and hard to inspect afterwards.
Not every control on that wall needs the cook's cooperation. A blocked tool call or a stripped output happens whether the model likes it or not, and those are the guardrails worth reaching for first. But a large part of what a safety harness actually carries is instruction-shaped — a safety prompt, a procedure the agent is supposed to follow — and an instruction-shaped control is exactly the kind that does nothing if the model never reads it. That is the half SafeEvolve is aimed at.
The attack SafeEvolve is defending against is the one the lethal trifecta describes: an agent that reads untrusted content, holds private data, and can act on the outside world. Hostile text arrives inside a tool result — exactly the surface covered in tools as attack surface — and the model obeys it as though the operator had asked.
SafeEvolve reads the agent's own failed runs and writes the fix into two places at once: a bounded, reversible edit to the harness, and a training signal for the policy. The loop runs on completed, on-policy trajectories — real episodes from the model being improved, not a recorded dataset from somewhere else — so the evidence describes failures this agent actually produces.
The harness half of the update is deliberately small. Trajectory-level safety evidence is converted into bounded component updates to two things: the safety prompt, and the hierarchical skills the agent follows when it works. Both come out as auditable artifacts. The policy half is a two-stage recipe: harness-use SFT first, which teaches the model to consult and obey those artifacts, then harness-augmented reinforcement learning with verifier-decomposed rewards, which shapes behaviour while the artifacts are in place. The second stage matters because a guardrail changes the environment the model is exploring in — training with the card on the wall is not the same as training without it and hoping.
Because the harness half of the fix is an artifact rather than a weight update, you can read it, diff it and revert it on its own — where a weight update can only be rolled back whole, as a checkpoint, and never read as a rule. That is the property worth stealing even if you never run the training half. A guardrail that arrived as a named, versioned edit has an owner, a date and an undo; a guardrail that arrived as "we did another alignment pass" has none of those, even though the checkpoint it shipped in is perfectly revertible. It is the same argument the policy enforcement step makes about keeping enforcement declarative, and the same reason fail-safe versus fail-open is a decision you want written down rather than emergent.
| What you change after a failure | What it fixes | What it costs |
|---|---|---|
| Patch the harness only | Changes and reverts on its own, with no training run; a hard enforcement point works whether or not the model cooperates | An instruction-shaped control — a safety prompt, a procedure — is left to a model that was never trained to consult it |
| Retrain the policy only | Changes behaviour itself, so the fix is not tied to one control point in the wrapper | The fix is diffuse in the weights — you can revert the whole checkpoint, but you cannot read the rule that changed or edit it on its own |
| Co-evolve both (SafeEvolve) | ~3× lower attack success rate on AgentDojo with Qwen3.5-4B, benign utility 59.79% → 61.86% (source) | Two training stages — harness-use SFT, then harness-augmented RL with verifier-decomposed rewards; and the result is one model on one benchmark, not a general law |
Putting numbers on the trade
The two headline numbers are measured over different populations, so walk them one at a time. Start with the attacks. Picture 1,000 episodes in which an injection is attempted (illustrative), against a baseline attack success rate of 24% (illustrative: the paper reports the ratio, not the baseline rate). A threefold reduction takes that to 8%, so 240 successful attacks become 80 and 160 attacks that would have landed no longer do.
Now the benign work, counted over its own 1,000 ordinary tasks (illustrative), because this is the number that often moves the wrong way — hardening an agent can make it timid, and safety is frequently paid for in refused work. Benign utility instead rises from 59.79% to 61.86%: 598 completed tasks become 619, 21 more. Attacks fell by two thirds while completed work went up, and that second half is the part you cannot assume you will get — the safety gain did not come out of the agent's usefulness.
One caveat worth carrying: 61.86% is still a long way from finishing every benign task, and the reported gain is a single 4B model on a single benchmark. Treat the direction as the finding and the magnitudes as one data point, not a rate you can plan capacity against.
The row of that table that actually distinguishes SafeEvolve is the last one, because it is the only one where the model has been trained to use the guardrail rather than merely shown it. Everything else is the choice engineers already make. If you run agents in production, the cheap version of this idea does not need reinforcement learning at all: write your guardrails as named, versioned artifacts, keep the runs that defeated them, and check — in shadow mode, before anything ships — whether the model behaves any differently when the artifact is present. If it does not, you have a rule card nobody reads.
Goes deeper in: Agent Engineering → Layered Guardrails → Defense-in-Depth for Agents
Related explainers
- HarnessBridge — the learned agent harness — the other half of the same argument: what happens when the harness itself is trained end-to-end instead of hand-written.
- Regression control in continual agent optimization — why repeated harness edits wash out without a guard against changes that only help the newest task.
- kNNGuard — the activation-space guardrail — a guardrail that lives inside the model's activations rather than in the wrapper around it.
- CAPTURE — counterfactual memory auditing — the same "gate the write on independent evidence" instinct, applied to what an agent is allowed to remember.