SparseDecoding paper — Decode-time calibration for LLM pruning — What does it mean?

The news. On October 8, 2026, researchers from Westlake University and Ant Group posted SparseDecoding (arXiv 2610.12327). It changes one input to existing pruners — where the calibration activations come from — and adds an N:M sparse matrix-vector GPU kernel for decode. On the WritingBench long-form writing benchmark, Qwen3-32B pruned to 2:4 sparsity scores 5.31 with decode-time calibration vs 2.99 with C4 calibration (the unpruned model scores 6.48), and the kernel reaches up to 1.48× end-to-end decode speedup on A100 GPUs. Read the paper →

SparseDecoding calibrates a pruner on activations from the dense model's own generated tokens, so its weight-removal decisions reflect what decode actually feeds each layer. To see why that matters, picture buying a lighter running shoe. The shop makes it lighter by cutting material away, and it decides where to cut by fitting the shoe to your foot. If that fitting happens while you stand still on the store's measuring board, every cut follows the shape of a resting foot. Then you go for a run. A few miles in, your foot is swollen and flexing in ways the store never saw, and the shoe pinches exactly where material was removed. SparseDecoding argues that one-shot LLM pruning has been fitting the shoe in the store.

Here is the store. Pruning zeroes out weights so a layer stores and reads fewer numbers. Training-free pruners such as SparseGPT work one layer at a time: run a small calibration set through the model, record the inputs each linear layer receives, then remove the weights whose loss changes that layer's output least on those inputs and adjust the survivors to compensate — the same Hessian-based recipe GPTQ uses for quantization. The calibration inputs are the foot the shoe is fitted to, and the usual foot is fixed text such as C4, fed with teacher forcing. Real generation is not teacher-forced: in decode, every new token is conditioned on tokens the model itself just wrote, so a small pruning error at one step changes the context of the next, and the activations drift away from anything calibration saw.

The paper measures that swelling. On Llama-3.1-8B, the gap between the dense model's activations and its pruned copy's grows fast over the first decoding steps and then levels off — every layer reaches 90% of its plateau by step 12 and stays there for hundreds of steps, much like a foot that swells in the first miles and then holds its size. The fix is to fit on the treadmill. SparseDecoding gives the dense model task prompts, lets it write its own continuations, records each linear layer's inputs only at decode steps (prefill activations are discarded), and passes those to the unchanged pruning solver. Same SparseGPT, same sparsity, the same 1M calibration tokens as the C4 baseline — only the foot changes. It even has to be your own foot: pruning Qwen3-14B on Qwen3-32B's generations scored 5.23, against 5.35 on its own generations (50% unstructured sparsity, where zeros may fall anywhere).

Scores are WritingBench overall quality on a 1–10 scale, judged by an LLM, with the same SparseGPT backend in both columns. The gain is real but uneven, and no row reaches dense. Qwen3 at 2:4 recovers more than two points; the Llama models gain about 0.2. Removing half the weights still costs quality — a better fitting shrinks the pinch, it does not remove it.

Why the pruned model was not faster

Fewer weights should mean faster decode. At batch one, each linear layer multiplies its weight matrix by a single vector, so decode sits on the memory-bound side of the roofline: step time is mostly the time to stream weights out of HBM. Half the weights should be close to half the bytes. But the standard 2:4 path (NVIDIA cuSPARSELt on Sparse Tensor Cores) is built for matrix-matrix work. On Llama-3.1-8B on an A100, the paper measures that path at 1.31–1.46× faster than dense in prefill, but only 0.85–0.87× of dense speed in decode — the pruned model generates text more slowly than the unpruned one.

SparseDecoding adds its own sparse matrix-vector kernel built on two choices. Bitmask indexing: instead of storing each kept weight's column number, it stores one bit per input column, packed 32 to a word; the set bits line up with the stored weights in order. Fixed-step traversal: at 50% N:M sparsity, every aligned 32-bit word holds exactly 16 set bits, so every row takes the same number of loop steps — known at compile time, unrollable, and uniform across GPU threads, so no thread waits on a neighbour with a longer row (an uneven-work problem closely related to warp divergence).

Hold sparsity at 50% and count the index bits that travel with the weights. A 32-column slice of one row keeps 16 weights. With int32 column indices, that is 16 × 32 = 512 bits of metadata; with uint8 offsets, 16 × 8 = 128 bits; with a bitmask, one 32-bit word — 2 bits per kept weight, 16× less than int32 and 4× less than uint8. Now add the weights themselves at FP16 (illustrative — the paper states only the metadata figures): the 16 kept weights are 256 bits. With int32 indices the slice costs 768 bits, more than the 512 bits of the dense slice it replaced; with the bitmask it costs 288 bits, ~56% of dense. Measured end to end on A100s with GPT-Fast (a compact PyTorch inference codebase), Llama-3.1-8B goes from 96.26 to 136.30 tokens/s at 2:4 (1.42×), and Llama-3.3-70B reaches 1.48× with a 16:32 pattern. That stays below the byte count alone, because only the pruned linear layers get lighter.

The idea outlives this paper: calibration data should look like the workload you will actually run, and for generation that workload is the model's own tokens. The same question applies to quantization calibration, which is also commonly done on fixed text. Limits worth knowing: results are in non-thinking mode on A100s; the kernel targets batch-one and small-batch decode, where layers are matrix-vector products; and collecting calibration data now means running generation, which is slower than one teacher-forced pass over stored text.

Goes deeper in: LLM Internals → Quantization → Modern Methods