What is the ShallowDefend Assess, Decide, Respond format?

It is a response format in which a model first assesses the user's intent, then commits to an explicit decision (COMPLY or REFUSE, plus SAFE-REDIRECT in the judge variant), and only then writes its reply conditioned on that decision. The decision is trained with reinforcement learning on the final, payload turn of a conversation.

Why does putting the decision first help against multi-turn attacks?

Safety alignment is shallow: a safety-trained model differs from its base model mostly in its first few output tokens. A multi-turn attacker steers those opening tokens toward compliance. Writing the decision in that span makes it a point the defender controls and a rule can check.

How is the self-verified reward different from an LLM-as-judge reward?

The self-verified reward compares the committed label with the known nature of the training conversation, which was grown from a harmful or benign seed prompt. It needs no judge and no annotation. The judge variant scores the reply with a second model over a graded decision space.

How does this relate to guardrails in an agent harness?

It is one layer of defense in depth. Because the decision is a readable token span, a harness can log and gate on it, but it still sits alongside input filters, output filters and capability scoping.

Agent· 2026-11-03

ShallowDefend paper — Assess, Decide, Respond safety commitment — What does it mean?

TL;DR

What is it: The ShallowDefend paper (arXiv, 29 October 2026) trains a chat model to assess the user's intent, commit to a parseable COMPLY or REFUSE decision, and only then write its reply, a format called Assess → Decide → Respond.
Why it’s needed: Multi-turn jailbreaks hide a harmful request across innocent-looking turns, and safety training mostly lives in the first few output tokens; writing an explicit decision exactly there turns that weak spot into a control point that a plain string rule can check.
vs previous: Earlier defenses (fine-tuning on attack dialogues, preference optimization, judge-scored reinforcement learning) show the model what safe replies look like and need an oracle model in the loop, while here the decision is a token span, so the reward is a string comparison with no judge.

Jargon

Multi-turn jailbreak: An attack that spreads one harmful request over several chat turns so that each message looks harmless on its own. The harmful ask arrives only in the last turn.
Shallow safety alignment: The finding that a safety-trained model differs from its base model almost only in its first few output tokens: it learns to open with a refusal, and if the reply opens compliantly the base behavior returns.
Payload turn: The final turn of an attack conversation, where the harmful content is actually requested. ShallowDefend applies its reinforcement-learning signal only to this turn.
Prefilling attack: Forcing the reply to start with an agreeable prefix such as "Sure, here is", which defeats a refusal that lives in the opening tokens.
Verifiable reward: A reward computed by a rule instead of a model, as in RLVR (Reinforcement Learning with Verifiable Rewards). Here the rule compares the committed label with the conversation's known nature.
LLM-as-judge: Using a second language model to score a reply, for example as safe or jailbroken. It can disagree with other judges on individual samples, which is the weakness the rule-based reward avoids.
Per-token KL divergence: A measure of how far the trained model's next-token probabilities moved from the base model, computed at each position of the reply. It shows where training changed the model.
Safe-redirect: A third decision option in the judge-scored variant: deflect a harmful request toward safe help without an outright refusal.

The news. On October 29, 2026, a team including researchers at UC Davis and Microsoft Research posted ShallowDefend, a method that trains a model to write an explicit safety decision before every reply. They report that, across two model families and fourteen benchmarks spanning safety, utility and helpfulness, it gives the strongest multi-turn safety in a controlled comparison, at a modest cost in helpfulness. Read the paper →

Picture an airport gate where a traveler arrives with a long, friendly chat history. The usual defense is a gate agent who has learned to say a polite "I'm sorry, I can't" in the first words of an answer. An attacker simply keeps the conversation pleasant until the natural first words of the next answer are "Of course", and the refusal is gone before anything harmful has been said. That is the paper's account of shallow alignment: safety behavior sits in the opening tokens, and a multi-turn attacker steers those tokens.

ShallowDefend keeps the gate agent's habit but makes it explicit. Before the agent speaks to the traveler at all, the agent reads the whole file, then stamps the pass with a decision, and only then answers in a way that matches the stamp. The stamp occupies exactly the opening positions that the attacker was steering, so the weak spot becomes the place the defense is installed.

Mechanically, each turn is one generation with three labeled parts: an assessment of the user's real intent and whether fulfilling the request could cause harm, a decision line (COMPLY or REFUSE, plus SAFE-REDIRECT in the judge variant), and then the reply. The paper names three properties of the decision span. It is externalized (a token span that a rule can read), binding (it comes before the reply in the same generation, so changing it changes the reply) and early (it sits where shallow alignment concentrates).

Reinforcement learning rewards only the payload turn, where the attack delivers its request and the defense must hold. Training conversations are grown from seed prompts whose nature is known in advance, so each one carries a free label, and the self-verified reward is a string match: refuse on conversations grown from harmful seeds, comply on benign ones. Where a judge model is available, the same format also supports a judge-scored reward over the graded decision space.

For an agent builder this is one layer in a defense-in-depth stack, next to input filters that screen what enters and capability scoping that limits what a fooled model can do. Its distinguishing feature is that the decision is written in the model's own output, so a harness can log it, read it and gate on it, much as a guardrail classifier like kNN-Guard reads a signal from outside.

Defense	What the model is shown	Oracle model needed?
Fine-tuning on adversarial dialogues	Safe replies to attack conversations	Yes, to say what a safe reply is
Preference optimization	Pairs of safe and unsafe replies	Yes, a preference labeler
Judge-scored reinforcement learning	A score for each reply	Yes, a judge or guard model
Inference-time intent prompting	A prompt asking it to analyse intent first (no training)	No, but nothing is learned
ShallowDefend, self-verified variant	A committed label that a rule checks	No, the label is compared with the known nature of the conversation

Here is the reward arithmetic on a tiny batch (illustrative). Take 4 training conversations: 2 grown from harmful seeds and 2 from benign seeds. On the payload turn the model commits REFUSE, COMPLY, COMPLY, REFUSE. The rule scores 1 for a correct label and 0 otherwise, so the batch gets 1, 0, 1, 0. That is 2 of 4 labels correct, a mean reward of 0.5, computed with 0 judge calls. A judge-scored reward would need 4 calls to a second model, one per reply, and would inherit that model's disagreements. The learning signal also lands on a short span: if the assessment and decision line take about 50 tokens of a 350-token reply (illustrative), roughly one token in seven carries the part that the paper's analysis says sets the whole trajectory.

Two limits are worth keeping in mind. The decision is a trained behavior, not a guarantee, so it belongs in a stack with filters and capability limits rather than in place of them. And the self-verified reward needs training conversations whose nature is known in advance, which is easy for seed-grown attack dialogues but is a data-construction step you must own. The paper reports that the decision acts as a causal control point and that training concentrates on the brief deliberation span; the numbers behind the safety and helpfulness trade-off are in the paper's results tables.

Goes deeper in: Agent Engineering → Layered Guardrails → Defense-in-Depth for Agents

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based