ASTC paper — Agentic soft-token compression vs hard-token context rewriting — What does it mean?

The news. On October 30, 2026, a team led from Shanghai Jiao Tong University posted a paper on Agentic Soft-Token Compression (ASTC). It needs no training: using a frozen Qwen3.5-9B agent on three long-horizon benchmarks, the authors report accuracy close to the uncompressed agent with up to 50.68% fewer end-to-end attention pairs, while trained compressors cost 15.4–103.7× more attention pairs per successful task. Read the paper →

Picture a navigator keeping an expedition journal. Every day adds pages of raw notes: instrument printouts, route logs, weather tables. By day thirty the journal is too heavy to reread before each decision. The usual fix is to hire a second writer to rewrite the journal as shorter text. That writer never walked the route. They see only the text, so they must guess which pages the navigator has already absorbed, and a detail that looks unimportant today can decide an action days later. They must also write in plain words, which loses anything spread across many lines. Some things cannot be paraphrased at all: a coordinate, a locker code, a file path. Change one character and the later action fails.

An LLM agent keeps the same journal. Each step appends instructions, earlier decisions and tool results, and the paper's analysis of real agent trajectories found that tool observations are the largest and fastest-growing share of that context. A tool observation mixes prose and code that the model has probably digested with exact values that a later action may copy verbatim. This is the scarce-context problem, and the usual answer is a summarization call, one of the standard context fixes. ASTC asks a different question: why not let the navigator fold their own pages?

ASTC works on hybrid models that mix attention layers with Gated Delta Network (GDN) layers. A GDN layer keeps a fixed-size memory matrix and updates it for each token by only the part of the token's value that the memory could not already predict. If the memory already predicts a token, the update is small; a surprising token changes the memory a lot. So the model's ordinary forward pass already computes a running answer to "is this new to me?". An external compressor has to learn to guess that answer from text. ASTC reads it directly.

At a chosen boundary layer, for each tool-output span, ASTC runs two checks on every position that is not protected:

• Innovation: how much of this token's value the memory failed to predict. A small value means the memory largely predicted the token, which makes it a candidate for merging.
• Readability: after the whole observation, can the token's value still be read back out of the memory with a fixed, untrained read-out? Being predicted once is not enough, because a later update might have overwritten it.

A position qualifies only if it passes both. Qualifying positions are split into at most a fixed number of groups, in order, and each group's hidden states are averaged into one soft token that stays in hidden space and is never turned into words. A group is merged only if the average stays within a relative-error threshold. A protection mask keeps recognized exact values and the action boundary as hard tokens (the authors note it may miss some exact fields). Early layers still process every original token; the layers after the boundary see the shorter mixed sequence.

The saving does not stop at the step where compression happens. The shorter downstream KV cache is kept, so every later query attends to fewer keys without another compression decision. The paper's Theorem 4.1 counts this in attention pairs. With A full-attention layers after the boundary, r removed positions, a prefill of n tokens and N later queries, the reduction is A × r × ((2n − r + 1) / 2 + N).

A worked example with illustrative numbers (not from the paper's experiments): hold n = 10,000 tokens, r = 1,000 removed positions and a single full-attention layer fixed. The prefill term is (2 × 10,000 − 1,000 + 1) / 2 = 9,500.5, so that layer saves 1,000 × 9,500.5 ≈ 9.5 million attention pairs in the current prefill. Each later query saves another 1,000 pairs, one per removed position, so after about 9,500 later queries the reuse has saved as much as the prefill did, roughly 9.5 million more pairs, and it keeps growing. The paper stresses that this counts attention pairs, not wall-clock speed, and that selection and pooling cost must stay below the work avoided.

The paper's Table 1 compares methods on AppWorld with a frozen Qwen3.5-9B acting model. Workload is relative to the uncompressed agent (100%) and includes the calls the compressors make themselves.

Read the results at the right size. On AppWorld, the paper says ASTC is the only evaluated method that improves accuracy over the uncompressed agent while lowering decode-KV, attention-pair and generated-token workload. The largest saving, 50.68% fewer attention pairs, comes on OfficeBench, where ASTC scores 47.37% against 48.42% uncompressed: close, not higher. A simple prompted summary scores higher there (52.63%) but uses more decode KV and more generated tokens. On the larger Qwen3.8-27B backbone, attention pairs fall 35.57% while generated tokens rise 3.82%.

Three limits matter. The method needs a hybrid GDN–attention model, because a plain attention model has no GDN memory state to read. The main experiments use one frozen 9B backbone. And the authors call their selector-versus-random comparison "suggestive rather than conclusive". The idea to keep is architectural: when the model already tracks what it has absorbed, compaction can be a read of that state instead of a second model's guess.

Goes deeper in: AI Agents → Context Engineering → The 4 Fixes