HyScale sparse-attention training — Selection without a score table — What does it mean?

The news. On October 23, 2026, researchers from the University of Edinburgh and Tencent Hunyuan posted HyScale, a training system for learned sparse attention. They report a 2.07× speedup for GLM-5.1 training on 1M-token sequences over Megatron-LM on 512 H20 GPUs, and 2.01× and 1.57× for the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next at 1M tokens on 32 H800 GPUs. Read the paper →

Picture a contest with a million contestants and thousands of judges, where each judge must name the 2,048 best contestants for their own shortlist. The wasteful way is for every judge to write all one million scores on a giant scoreboard, then read the whole board back to find the best 2,048. That is how current systems run selection in learned sparse attention: one kernel writes the full score table to GPU memory, and a second top-k kernel reads it back. The indexer's scores are the same kind of work as computing attention scores, only cheaper per token, and the table they fill lives in HBM.

The paper's key observation is that the output of selection is a set of indices, not an activation. Sparse attention reads only the selected entries, and the indexer's own loss reads only the scores at those entries. Because top-k is not differentiable, no gradient flows through the choice, so the backward pass never needs the scores of the other keys. Nothing downstream reads the scoreboard, which means nothing forces the system to build it, or to rebuild it later. Two techniques follow from that: HyTopK removes the scoreboard, and selection replay removes the repeat judging.

The same move shows up in FlashAttention, which also refuses to write a huge intermediate matrix to memory and instead works on pieces that stay on chip (FlashAttention: The Solution). HyTopK applies that idea to the indexer's score table.

HyTopK is the judge who sets a pass mark first. It samples evenly spaced keys from the query's own context, between 4,096 and 8,192 of them, and sets the threshold at the sample score that stands for roughly the 3k-th best key. The scoring kernel compares each score with the threshold on chip and writes only the keys that pass into a small candidate buffer. Because the threshold lets about three times k keys through, it rarely rises above the true k-th best score, so every true top-k key survives and an exact selection over the candidates returns the exact top-k. The authors argue the threshold does not have to be a guaranteed lower bound; it only has to fail rarely and be cheap to repair. A check kernel flags any query whose candidates overflowed the buffer or number fewer than k, and small recovery kernels rescore just those queries over their whole context. The paper reports that the case of more than 64 failed queries in one selection, which falls back to rescoring everything, never occurred in its training runs. To stay bitwise reproducible despite atomic writes into the buffer, HyTopK resolves ties at the k-th score with a radix select over positions and orders the k kept entries with a radix sort.

Training adds a second cost that inference does not have. With full activation recomputation, each layer's backward pass first reruns that layer's forward pass, and the rerun includes selection. In the contest, that is calling every contestant back the next day to be scored again, while yesterday's shortlist is still on file. Selection replay keeps the k indices per query from the forward pass and, during recomputation, rescores only the k entries they point to. Existing selective recomputation cannot do this because it decides per operator whether to keep an output or rerun the whole operator, and so it cannot rerun scoring for just k entries. The table below shows the policies the paper compares.

Here is the paper's GLM-5.1 setting at 1M tokens, with L = 1,048,576 keys, n = 8,192 queries per GPU and k = 2,048, worked through with the paper's formulas. The score table is 8,192 × 1,048,576 × 4 bytes, which is 32 GiB per layer on each GPU, while the selected indices are 8,192 × 2,048 × 4 bytes, which is 64 MiB. That is 512 times smaller, because top-k keeps only 0.2% of the scores. For HyTopK's threshold, the sample size is M = min(max(4,096, 1,048,576 / 128), 8,192) = 8,192, so each sample stands for 128 keys, and the threshold is the ⌈3 × 2,048 × 8,192 / 1,048,576⌉ = 48th best sample, which stands for 48 × 128 = 6,144 keys, three times k. In expectation, each query's candidate buffer therefore receives about 6,144 keys instead of a row of 1,048,576 scores, with the rare query that falls short or overflows sent to recovery. On replay, keeping only the indices still forces a rescore of all 1,048,576 keys per query, while index replay rescores 2,048, which is 512 times fewer scoring operations; the paper reports that this rescoring costs at most 1.6% of a training step.

Two limits matter. The 2.07× figure is end to end for one model at one scale, and the paper's ablation attributes most of the speedup to HyTopK and selection replay, with the remainder coming from hiding context-parallel communication behind selection and from a sparse kernel that handles two adjacent queries together. And the idea depends on selection being non-differentiable and on training that recomputes activations; it does not make the indexer's scoring itself cheaper, it only stops the system from paying for it twice and storing it once.

Goes deeper in: LLM Internals → Self-Attention → Computing Attention Scores