The Kernel Reality Check — Kernelized linear attention vs FlashAttention — What does it mean?
The news. On 17 November 2026, a four-author team posted The Kernel Reality Check (arXiv 2611.22947). It profiles two state-of-the-art kernelized methods, Hedgehog and DiJiang, against FlashAttention v2.3.6 on one A100 GPU, with every method in the same harness and the kernelized ones on Triton-optimized FlashLinearAttention kernels. The models are GPT-2 language models up to 774M parameters and vision transformers up to ViT-L. The paper then distills pretrained DeiT and DINOv2 models into kernelized versions to measure what the swap costs in accuracy. Read the paper →
Picture two drivers leaving the same garage for the same address. One takes the city route: on the map it is much shorter, but there is a stoplight on every block. The other takes the highway: more miles, no stops. Big-O counts the miles; a GPU charges for the stops. That is the paper's central finding, and it maps directly onto how each method runs.
Kernelized attention shortens the map. Softmax attention scores every query against every key — N2 scores for N tokens. A kernelized method replaces the exponential score with a dot product of two feature vectors, φ(q)·φ(k). That lets it add every key-value pair into one small summary matrix S, once, and answer each query from S. On paper, the cost now grows with N, not N2.
The stoplights appear in language models. Under causal masking, token 9 may only see tokens 1 to 8, so there is no single S: each position needs its own running total, Si = Si−1 + φ(ki)viT. Each running total depends on the one before it, so the work becomes a chain of dependent memory accesses instead of one large parallel matrix multiply. The authors note that on modern accelerators memory latency typically dominates arithmetic — the memory-bound side of the roofline — which is exactly where a shorter map stops helping.
FlashAttention keeps every one of the N2 miles but removes the stops. It splits Q, K and V into tiles that fit in on-chip SRAM, computes each tile's scores and softmax there, and never writes the full N×N matrix back to HBM. The arithmetic is large and dense, and it stays on the fast part of the chip.
Measured with every method in one harness, the highway arrived first almost everywhere. FlashAttention trained faster and with less memory than both Hedgehog and DiJiang at every tested model size, and in vision transformers it was the fastest and lowest-memory method at every image resolution. Vision is the telling case: a ViT is bidirectional, so there are no causal stoplights and one shared S serves every query — yet FlashAttention still won all six model-and-phase combinations, and the authors say a full roofline analysis would be needed to explain the rest of the gap. The measured range ends at 16K tokens; the paper places the crossover for wall-clock time only at very long sequences, 32K tokens and up.
Two published speed claims also reversed. Hedgehog's reported speedup of nearly 5× over FlashAttention v1 at 32K tokens, measured on a single attention layer, became a 3.5× slowdown in prefill (the single forward pass over the prompt) at 8,192 tokens against FlashAttention v2 inside a full GPT-2 model — still 1.4× slower after moving Hedgehog onto Triton kernels.
The one leg the city route wins is carrying luggage on a long trip. During causal language-model generation, linear attention's backpack never grows: Hedgehog's reference implementation held about 533 MB however many tokens it produced, while FlashAttention tows a KV cache that grows with every token. Prefill memory shows the same split — DiJiang used 2.6–3.1× less than FlashAttention at 16K tokens — although FlashAttention was still the fastest prefill method at every model scale. Generation speed is a narrower win: Hedgehog's reference implementation produced 16K tokens on GPT-2 Small in 79.5 s against FlashAttention's 90.0 s, but its Triton-kernel version took 91.0 s.
Here is the whole trip for one car: GPT-2 Small on one A100, with every number measured in the paper's Table 6. Training at 4K tokens, FlashAttention took 111.6 ms and 3,264 MB per step; DiJiang on Triton kernels took 131.0 ms and 4,201 MB — about 17% slower and roughly 0.9 GB heavier, despite doing asymptotically less work. Now make the trip 4× longer, 16K tokens: softmax's score count grows 42 = 16× while the kernelized state updates grow only 4×, so on the map the city route's lead quadruples. Yet in a 16K-token prefill, FlashAttention still finished first, 215.2 ms vs 304.8 ms. The luggage tells the other story: after generating 16K tokens, peak memory was 2,991 MB for FlashAttention, which carries the growing KV cache, against 533 MB for Hedgehog — about 5.6× more.
| GPT-2 Small, one A100 | Train, 4K tokens (ms / MB) | Prefill, 16K tokens (ms / MB) | After generating 16K tokens (MB) | Source |
|---|---|---|---|---|
| Softmax (naive PyTorch) | 315.7 / 15,051 | 2,961.9 / 28,680 | 3,016 | Table 6 |
| FlashAttention v2 | 111.6 / 3,264 | 215.2 / 2,886 | 2,991 | Table 6 |
| Hedgehog (Triton FLA kernels) | 130.6 / 4,930 | 304.0 / 1,583 | 481 | Table 6 |
| DiJiang (Triton FLA kernels) | 131.0 / 4,201 | 304.8 / 1,256 | 507 | Table 6 |
Speed is half the paper. The other half asks whether the shorter route at least reaches the same address — and for current methods it does not. Distilling pretrained DeiT image classifiers into kernelized versions left top-1 accuracy gaps of 1.8–2.5 points for Hedgehog and LARA and 7.3–10.1 points for DiJiang, and the gaps did not shrink as the model grew. The authors' LARA++ closes most of that gap: it approximates softmax by importance sampling, groups similar queries with a greedy k-center clustering step, and uses each group's proposal mean as a fixed feature, reaching 82.1% against softmax's 82.9% on DeiT-B/384. But the clustering alone takes 43–60% of its forward time on DINOv2, and in the paper's efficiency sweep its memory advantage appears only from about 64K tokens. Fixing the approximation did not fix the speed.
The lesson outlives these two methods: a complexity class predicts how cost grows, not which kernel finishes first on a given chip. Before trusting a speedup chart, check its baseline. The paper found that DiJiang's flat generation throughput — about 174 tokens/s from 128 to 16,384 tokens — was also true of FlashAttention with a KV cache, at 193 falling to 182 tokens/s, because at these lengths each decoding step is dominated by streaming the model weights. The authors point instead to hybrids such as BASED and GLA, which combine linear global attention with exact local attention, as the more promising direction.
Goes deeper in: GPU & CUDA → Operator Fusion & FlashAttention → IO Complexity: Why It's Faster
Related explainers
- I/O-optimal approximate attention vs FlashAttention — the other route: approximate attention that cuts data movement itself, with proven lower bounds
- Gated DeltaNet-2: decoupled erase and write gates — how modern linear attention is trained from scratch rather than swapped into a softmax model