What are Heads of Security (HoS)?

HoS is a prompt-injection defense that adds a small learned gate to each attention head. The gate lowers how much the head attends to tokens marked untrusted, such as tool output, so the model is less likely to follow instructions hidden in text it only needs to read.

Why does gating attention help against prompt injection?

Research cited by the paper finds that specific heads shift their attention onto injected instructions during an attack. Closing those heads' access to untrusted tokens reduces one route by which injected text steers the answer, while other heads still read the content.

How is HoS different from SecAlign?

SecAlign fine-tunes model weights with preference training. HoS trains only the gate scalars on the same preference data and keeps base weights frozen. In the paper's tests this kept few-shot accuracy close to the base model, where SecAlign dropped sharply in several settings.

Does HoS make an agent immune to prompt injection?

No. The authors say it enforces a trust boundary but does not decide what to trust, applies only to dense attention, and cannot rule out a strong adaptive attacker. It is one layer in a wider defense.

Agent· 2026-11-02

Heads of Security — per-head attention gates — What does it mean?

Jargon

Prompt injection: An attack where text planted in external content (a web page, an email, a tool result) is mistaken by the model for a real instruction from the user.
Instruction–data separation: The rule that authorized instructions may steer the model, while external content may be read and analyzed but never treated as a command.
Attention head: One of the parallel attention computations inside a transformer layer. Each head scores every earlier token and takes a weighted mix of their values. How multi-head attention works.
Logit: The raw score a head gives a token before softmax turns all the scores into weights that sum to 1.
Attention sink: A token, often the first one, that heads park spare attention on when nothing else is relevant. GPT-OSS uses one, so HoS gives the sink its own key-only channel.
ASR (attack success rate): The share of attack attempts that make the model do what the attacker wanted. Lower is better for a defense.
Adaptive attack: An attack tuned against the specific defense being tested, as opposed to a fixed list of known attacks. It is the harder and more honest test.

The news. On October 27, 2026, researchers at Imperial College London, the Tübingen AI Center and partners posted Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness. They tested it on Llama-3.3-70B-Instruct and GPT-OSS-20B and report that RL-Hammer, a black-box attack, reached 97% success on GPT-OSS-20B and 85% on a SecAlign-defended copy, but only 10% on the HoS copy (the highest rate among the RL-Hammer attackers they trained). Read the paper →

Picture a team of readers at a desk. The boss left one sticky note, and a pile of incoming mail sits next to it. Each reader has a fixed budget of attention to split across everything on the desk, and one letter in the pile says "ignore the boss and wire the money." The old fix was to retrain the whole team to distrust mail. Heads of Security does something smaller: it puts a dimmer on the lamp over the mail pile, one dimmer per reader, so each reader can still skim the letters but spends most of its budget on the boss's note.

In transformer terms, every attention head already scores each earlier token and turns those scores into weights with softmax. HoS first labels every token with a channel that says how far it is trusted: system and user text and the model's own output are trusted, tool output is untrusted, and chat-template tokens (including the attention sink) get their own channel. Then each head gets a small table of learned gate logits, one per pair of query channel and key channel, and the head multiplies the unnormalized weight of each key by a sigmoid of its gate. A gate near 1 leaves the head alone. A gate near 0 almost closes that channel to that head, the same trick causal masking uses when it adds a huge negative number to hide future tokens. If every token shares one channel, the gates cancel in softmax and the model behaves exactly like the base model.

Training touches only those gate scalars. The paper follows the SecAlign++ recipe: preference pairs where the user asked for one thing, an injected instruction in the untrusted segment asked for another, and the preferred answer follows the user. Because the model can only change how much each head looks at each channel, it has far less room to learn shortcuts tied to the training format. The authors argue this is why HoS keeps few-shot accuracy while SecAlign loses it: on GPT-OSS-20B, 5-shot MMLU-Pro is 68.3% for the base model, 67.2% with HoS, 59.4% with SecAlign and 52.1% with ASIDE. The gates sit inside the multi-head attention the model already has, so no new layers are added.

Defense	What it changes	GPT-OSS-20B 5-shot MMLU-Pro	Highest RL-Hammer ASR
No defense	Nothing	68.3% (Table 2)	97% (Table 1)
SecAlign (SecAlign++ recipe)	Fine-tunes extra weights with DPO preference training	59.4% (Table 2)	85% (Table 1)
ASIDE	Rotates the embeddings of data tokens	52.1% (Table 2)	not in the table the article cites (paper)
HoS	Learns per-head gates on attention logits	67.2% (Table 2)	10% (Table 1)

The paper also looked inside the learned gates. Most heads learned to look less at untrusted keys, which is the expected direction for heads that follow instructions. A minority did the opposite and look more at untrusted content. On an email benchmark (LLMail-Inject, a detection test separate from the attack-success numbers above), keeping only those "blue" gates raised the rate of correctly flagging an injection from 11.03% to 28.20%, while keeping only the "red" suppressing gates dropped it to 3.60%. So the model seems to split the work: some heads stop obeying the mail, and a few heads read it closely enough to notice it is a trap. Against a white-box attacker that was tuned against HoS itself, the attack needed roughly three times as much compute to reach 50% success as it did against the base model or SecAlign.

Here is the gate arithmetic with small numbers. These values are illustrative, not from the paper. One head sees 1 trusted instruction token and 10 injected tokens, all with the same raw score. With no gate, the injection holds 10 of 11 weights, which is about 91% of that head's attention. Set the untrusted gate to 0.1, and the ten injected weights become 10 × 0.1 = 1, so the split is 1 to 1 and the injection falls to 50%. Set it to 0.01 and the injection weight is 0.1 against 1, which is about 9%. The head never refuses to look at the mail, and it still ends up listening mostly to the boss.

The limits are stated plainly in the paper. Instruction–data separation is not an end-to-end defense: HoS enforces a trust boundary but does not decide which sources deserve trust, and some tasks (installing a package from a README) legitimately follow instructions found in data. The gates need standard dense attention, so they do not directly fit linear-attention models. And no finite test proves security against an attacker who moves second; the evidence is that the same attack budget works less well here. Treat it as one layer next to capability scoping and output filtering, not a replacement.

Goes deeper in: AI Agents → Security & the Lethal Trifecta → The Lethal Trifecta. For the other half of the defense, see cutting off a leg of the trifecta with capability scoping.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based