What is TriAttention?

A sparse mixed-precision attention method that assigns each query-key tile to skipped, 4-bit or 8-bit computation. Each head stores only two distance boundaries, w8 and w4, chosen by offline calibration.

Why combine skipping and 4-bit precision?

Some attention interactions tolerate low-precision scores but are harmful to remove. Pricing skipping and precision in one cost budget lets the allocator keep those tiles at 4-bit instead of choosing between dropping them and paying full precision.

How does it differ from using 4-bit attention everywhere?

A uniform 4-bit reference keeps every causal tile at low precision outside the protected tiles. In the paper it scores worse on perplexity, MMLU and 64K LooGLE-SD than the mixed policy, which skips far tiles and keeps near tiles at 8-bit.

Does the 7.88× speedup apply to normal serving?

Not necessarily. 7.88× is the best reported end-to-end speedup over Torch attention; against a BF16 baseline the best is 2.65×, and the 64K Llama example in the paper shows 1.62× throughput.

GPU· 2026-11-09

TriAttention paper — Sparse mixed-precision attention tiles — What does it mean?

Jargon

QK tile: A block of queries paired with a block of keys. The kernel computes their score matrix as one unit, and skipping a tile skips its score, softmax and value work together.
Prefill and decode: Prefill reads the whole prompt once. Decode then generates one token at a time and rereads the stored keys and values each step. See prefill vs decode.
4-bit and 8-bit QK: The query-key multiplication done on 4-bit or 8-bit integer codes plus a scale, instead of 16-bit numbers. Lower-bit arithmetic can be faster, but it has less precision and can introduce more error. See how numbers shrink.
Diagonal distance: How many blocks separate a query block from a key block. Distance 0 is the diagonal tile, where a query attends to its own block.
Online softmax: A way to add up attention one tile at a time by tracking a running maximum, a running sum and a running weighted value total, so tiles never need to be held together. See FlashAttention: the solution.
GQA: Grouped-query attention: several query heads share one set of keys and values, which shrinks the KV cache.
Hadamard transform: A fixed rotation applied to queries and keys before quantizing them. It spreads large values across dimensions so one outlier does not stretch the whole quantization range.
TPOT: Time per output token: the average delay between generated tokens during decode.

The news. On November 5, 2026, a University of North Texas and Saint Louis University team posted TriAttention on arXiv. The paper reports up to 7.88× end-to-end generation speedup over Torch attention and up to 2.65× over a BF16 baseline, with quality close to the original models. Read the paper →

Picture an editor with a 600-page manuscript who must check page 580 against everything before it. Reading every earlier page closely is the dense-attention bill; most of it buys nothing. The editor's real habit is by distance: the last few pages get a close read, the middle stretch gets a skim, and the far past is not opened at all, except page 1, which sets the terms and is always kept.

TriAttention turns that habit into a rule over tiles. For a query block and a key block that are Δ blocks apart, a head's pair of boundaries (w8, w4) decides the action: distances below w8 run at 8-bit, distances from w8 up to w4 run at 4-bit, and everything at or beyond w4 is skipped. The first key block and the diagonal tile are protected and always use 8-bit QK. The reason for mixing is the paper's central observation: some interactions tolerate low-precision attention scores while staying sensitive to being removed, so skipping them and rounding them are different decisions that share one error budget.

Two integers per layer and head replace a per-tile map, so policy storage does not grow with context length. The boundaries are chosen offline. Calibration scores each candidate pair by how much it changes the head's output after the attention output projection, normalized by the layer's reference output energy so layers are comparable. It also prices each candidate by the number of 8-bit and 4-bit tiles it keeps, with a measured factor γ for how much cheaper a 4-bit tile is than an 8-bit one.

A model-wide allocator then minimizes summed calibration error under one shared cost budget, so a head that is fragile to removal can keep more tiles while a head that is tolerant spends less. This is the precision-throughput tradeoff made per head instead of per model.

Approach	What it skips	Precision of what remains	Weak spot named by the paper
Dense attention kernels	Nothing	Full	Still evaluates every causal interaction
Structured sparse attention	Tiles outside a pattern	Full	Retained tiles still pay floating-point cost; more savings means dropping useful context
Uniform low-bit attention (all_int4)	Nothing	4-bit outside the protected tiles	Error lands in sensitive regions too
TriAttention	Tiles beyond w4	8-bit near, 4-bit in the middle	Needs offline calibration per model

The boundaries drive the GPU kernel directly. Each thread block reads its head's two integers, combines them with its query-block index, and writes a short list of key-block indices into shared memory, grouped as protected, 8-bit, then 4-bit. Skipped tiles never enter the list, so their score, softmax and value work disappears. Before this, the runtime centers the keys and applies a Hadamard transform to queries and keys, and prepares 8-bit and 4-bit formats with scales.

All groups feed one register-resident attention state, merged with the same running-maximum rule used in FlashAttention. Protected tiles use 16-bit values for the weighted sum; the other retained tiles use 8-bit probabilities and values with tile-local scaling. The quantized keys and values made during prefill are then packed once into a compressed prompt KV cache, so every decode step rereads less data. Because GQA shares storage across a query-head group, the cache keeps the union of the tiles the group's heads asked for.

A small illustrative case (illustrative numbers, not from the paper). Take query block 20, so 21 causal key blocks exist. Two are protected (block 0 and the diagonal), leaving 19 at distances 1 to 19. Choose (w8, w4) = (4, 12) and let a 4-bit tile cost γ = 0.6 of an 8-bit tile. Distances 1 to 3 give 3 tiles at 8-bit, distances 4 to 11 give 8 tiles at 4-bit, and distances 12 to 19 give 8 skipped tiles. The all-8-bit cost over the 19 eligible tiles is 19 units, while this policy costs 3 + 0.6 × 8 = 7.8 units. That is about 41% of the cost, roughly 2.4× cheaper on these tiles, while keeping 13 of the 21 causal tiles (including the 2 protected tiles). A more fragile head might get (8, 16) instead, spending more of the shared budget where its error is higher.

The paper's reported results come from a calibrated table reused across tasks. On a 64K prompt with Llama, it reaches 22.80 tokens/s against 14.06 for BF16 (1.62×), cuts TPOT from 26.64 to 10.06 ms, lowers peak allocation from 15.65 to 11.76 GiB, and reports 36.5% less energy per generated token. On quality, word perplexity is 12.50 against 12.49 for BF16, and MMLU is 61.32% against 60.66%. The full-context all_int4 reference is worse on perplexity (12.59) and MMLU (60.26%), and at 64K on LooGLE-SD it scores 44.82 against TriAttention's 47.94 and BF16's 48.40.

Treat the headline 7.88× and 2.65× as best-case numbers for the paper's settings, not a typical gain. The quality evidence is one model family and one calibration recipe, and the method needs an offline calibration pass before use; the architectural claim to take from it is the joint allocation, not a specific speedup.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Precision-Throughput Tradeoff

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based