What is attention-type decode partitioning?

It is splitting the decode step of LLM inference by the complexity class of its attention rather than by which operator is running. In SQD, the part that must index through the whole KV cache is separated from the part whose memory footprint stays the same size no matter how long the context grows.

Those two halves have opposite bottlenecks — one wants memory bandwidth and capacity, the other wants compute — so running both on the same accelerator means over-buying one resource to satisfy the other. Separating them lets each half run on hardware sized for its own limit, which is where the reported 31–56% tokens-per-joule gain comes from.

How is this different from prefill/decode disaggregation?

Prefill/decode disaggregation cuts between two phases of a request, and attention-FFN disaggregation cuts between two operator types. SQD cuts inside decode along a third axis: whether a step has to scan the full KV cache. That is why it can separate work the earlier seams leave bundled together.

Does it help a model with dense attention?

No. The gain depends on the model already using subquadratic attention — sparse, linear or sliding-window — because that is what creates the lopsided split between full-cache scanning and static-footprint computation. A dense-attention model has no such asymmetry to exploit, and it also needs heterogeneous hardware to place the two halves on.

LLM· 2026-09-14

SQD splits decode by attention type, not by operator — Attention-type decode partitioning — What does it mean?

TL;DR

What is it: SQD (SubQuadratic Disaggregation) is a serving scheme from a September 2026 paper, Rethinking Heterogeneous System Disaggregation for Subquadratic Attention, that splits one decode step in two by attention-type decode partitioning — the part that must scan the whole KV cache on one machine, the part whose memory footprint never grows on another.
Why it’s needed: Frontier models now lean hard on subquadratic attention, and the half of decode that must scan the entire KV cache has a completely opposite appetite from the half whose working set never grows — so putting both on one accelerator means sizing it for the worse of the two.
vs previous: Existing disaggregated serving cuts decode by operator type — attention on one machine, the feed-forward network on another — a seam that leaves full-KV scanning and fixed-footprint math sitting on the same side of the cut.

Jargon

KV cache: The keys and values already computed for every token in the context, kept so the model does not recompute them. It grows with the conversation. More in KV Cache.
Subquadratic attention: Attention variants whose cost grows slower than the square of the context length — sparse, linear and sliding-window attention are the common families.
Top-k selection: In sparse attention, the step that scores entries in the KV cache and keeps only the k most relevant ones. It has to index through the whole cache to decide.
Arithmetic intensity: FLOPs of math done per byte of memory moved. It is what decides whether a step is memory-bound or compute-bound. More in the roofline model.
FFN: Feed-forward network — the two dense projections that follow attention in every transformer block. Its weights are a fixed size regardless of how long the context is.
Disaggregation: Running different parts of inference on different machines instead of all of them on one, so each part gets hardware matched to its bottleneck.
Tokens per joule: Tokens generated per unit of energy. Where power rather than chip count is the binding constraint, it is this number — not raw throughput — that decides how much serving a deployment can do.

The news. On September 11, 2026, a paper titled Rethinking Heterogeneous System Disaggregation for Subquadratic Attention introduced SQD (SubQuadratic Disaggregation), which splits decode by quadratic versus subquadratic attention rather than by operator type. On an adjusted 8×B200 heterogeneous system proxy the authors report average tokens-per-joule gains of 53% on GLM 5.2, 31% on Nemotron 3 Ultra and 56% on Gemma 4 31B over the strongest GPU-only baselines. Read the paper →

Picture the archive at the end of a long day. The runner's job never gets easier: every errand means walking the full length of the stacks, and the stacks are longer this evening than they were this morning. The clerk's job never gets harder: however far the archive grows, the same few folders land on the same desk, and the desk holds the same amount of paper it always did. One errand, two people, opposite complaints — the runner needs aisles and legs, the clerk needs a sharp pencil.

Serving systems already split work across machines; the only question is where the cut falls. The familiar seam runs between prefill and decode — the phase that reads your prompt and the phase that writes one token at a time. A finer seam, and the one SQD measures itself against, runs between attention operators and the feed-forward network. Neither of those seams puts the runner on his own: prefill/decode keeps all of decode together, and the attention/FFN cut moves the paperwork away but still leaves the walk sitting with the attention it feeds. Either way, one machine ends up doing both the walking and the arithmetic, and it has to be good at two jobs that want opposite things.

That was tolerable while attention was dense, because then every layer had roughly the same appetite. It stopped being tolerable once frontier models started leaning on subquadratic attention to hold long contexts cheaply. Those variants do not make attention uniformly cheaper. They make it lopsided.

Prefill vs Decode on the Roofline

Same GPU, fundamentally different bottlenecks

The plot above is the textbook picture, and the textbook picture is exactly why decode gets treated as one thing: a single dot low on the memory ramp, around one FLOP per byte moved, nowhere near the compute ceiling. Arithmetic intensity is the horizontal axis — FLOPs done per byte moved, which is what separates memory-bound work from compute-bound work — and decode sits far to the left because each generated token drags a large amount of cached state across the bus to do very little math with it.

SQD's argument is that for a sparse-attention model, that single dot is an average of two points that are nowhere near each other. SQD cuts decode by attention complexity class instead: the half that must index through the entire KV cache on one side, the half whose working set never grows on the other. For sparse-attention models the split is top-k selection — the scan that scores the cache and keeps the k entries that matter — against top-k attention plus the FFN, which only ever touch the selected rows and a fixed block of weights. For linear and sliding-window models the same idea lands in a different place: the dense attention layers on one side, the subquadratic attention layers plus FFN on the other.

Where the cut falls	What lands on each side	What the two sides actually differ in
Prefill vs decode	Reading the prompt / writing one token at a time	Which phase of the request you are in
Attention vs FFN	Attention operators / feed-forward operators	Which operator you are running
Quadratic vs subquadratic (SQD)	Top-k selection / top-k attention plus FFN	Whether the step must scan the whole KV cache — and whether its footprint grows with context (paper)

Hold three things fixed and the gap stops being abstract (illustrative numbers): one layer, one request, a 128K-token context, on a model with 8 KV heads of 128 dimensions stored in FP16. That is 2 (a key and a value) × 8 heads × 128 dims × 2 bytes = 4,096 bytes of KV per token per layer, so this layer's slice of the cache is 128,000 × 4,096 ≈ 524 MB — and top-k selection has to index through all of it. Now set k = 2,048. The attention that follows reads 2,048 / 128,000 of those rows — roughly 8.4 MB — and the FFN behind it reads the same weight block whether the context is 8K tokens or 8M. Counted in KV bytes, the two halves of one decode step differ by about 62×, and with k held fixed, only the scanning half grows as the conversation gets longer. The FFN moves its weights on top of that, but those weights are the same size at 8K tokens and at 8M — they add a constant, not a slope. That is the whole point: memory cost is not one number here, it is two numbers with different slopes.

That difference is the entire lever. Put both halves on one accelerator and you buy for the worse of the two appetites; put each on a machine chosen for its own bottleneck and the bandwidth-hungry scan stops paying for compute it never uses. On an adjusted 8×B200 heterogeneous proxy the paper reports 31–56% more tokens per joule than the strongest GPU-only baselines — 53% on GLM 5.2, 31% on Nemotron 3 Ultra, 56% on Gemma 4 31B. A separate analytical model — of a next-generation heterogeneous system under a fixed power budget, not hardware anyone put a stopwatch to — projects 1.2×–1.5× tighter achievable latency and up to 3.6× higher throughput over attention-FFN disaggregation. Energy per token is a capacity-planning number more than a benchmark trophy: where power is the binding constraint, tokens per joule is what decides how many requests a deployment can hold.

The caveat is in the shape of the result. This is a placement win rather than a kernel win — it exists only if you have more than one kind of machine to place work on, and only for models whose attention is already subquadratic. A dense-attention model has no lopsided half to separate. A fleet built from one kind of accelerator has nowhere to put one. And each new seam adds a KV transfer that has to be paid for in interconnect — the abstract puts no number on that cost, though it does say the experiments produced findings about how to provision chips and interconnect for exactly this kind of system.

Goes deeper in: LLM Serving → Prefill/Decode Disaggregation → Full Disaggregation

Related explainers

NVIDIA Dynamo — Encode-prefill-decode disaggregation — the same instinct applied one seam earlier, at the front of the request
SubQ 1.1 Small — Subquadratic sparse attention — what the attention variant SQD is built around actually does
AMD ATOM — Prefill/decode disaggregation on ROCm — the baseline seam, and what it costs to move KV across it

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based