What is affine folding in FOSCAR?

It is a way to skip rebuilding each low-bit cached value as `s·q + z`. Because attention's matrix products are sums, the shared scale can be applied once to the other operand or to the partial result, and the offset becomes one correction term per group. The result is identical in exact arithmetic.

Why doesn't a 2-bit KV cache automatically make decode faster?

It cuts the bytes read from HBM, but every value still needs unpacking, conversion and rescaling before the Tensor Core can use it. That per-value instruction cost stays fixed as the bit width drops, so the bottleneck moves to instruction issue. The paper reports existing INT2 kernels using under 31% of H100 and 15% of B200 peak HBM throughput.

How much faster is FOSCAR?

On the paper's setup, INT2 decode attention runs a geo-mean **3.36×** faster than FlashInfer BF16 on H100 and **2.67×** on B200; INT4 runs 2.83× and 2.30× faster. Against B200's native NVFP4 path the gain is 1.16× for INT2 and 1.00× for INT4.

Does affine folding change model accuracy?

Affine folding adds no error of its own in exact arithmetic: measured against the dequantized cache, its kernel error is about 1.7 × 10⁻³, the same as BF16 decode. Any accuracy loss from using a low-bit cache comes from the quantization itself, so it still needs a good quantization recipe such as rotations or full-precision recent-token windows.

When does affine folding stop helping?

It performs `2·h_q / g` scale-and-offset operations per stored value, where `h_q` is query heads per KV head and `g` is the group size. It breaks even at `g = 2·h_q`, so it wins whenever groups are larger than twice the GQA ratio, which is true for typical groups of 32 or more.

GPU· 2026-11-10

FOSCAR speeds up 2-bit KV decode — Affine folding of KV dequantization — What does it mean?

TL;DR

What is it: FOSCAR (arXiv 2611.09722, November 2026) is a decode-attention kernel for 2-bit and 4-bit KV caches built on affine folding: it never rebuilds each cached value as s·q + z, and instead applies the shared scale and offset once per group, to the other matrix operand or to the matrix-product result.
Why it’s needed: A packed 2-bit KV cache reads 8× fewer bytes than BF16, but existing kernels spend so many GPU instructions unpacking and rescaling every value that decode may fail to turn the memory saving into a comparable speedup; folding removes most of that per-value work so the memory saving finally turns into speed.
vs previous: Earlier low-bit kernels such as KIVI, BitDecoding and OSCAR dequantize each value before the Tensor Core multiply, which makes instruction issue, not memory bandwidth, the bottleneck; FOSCAR keeps the same math but moves the affine step out of the inner loop.

Jargon

KV cache: The stored keys and values of every earlier token, read again on every decode step so the model does not recompute them. See the KV cache solution.
Decode attention: The attention computation for one newly generated token. It does little math per byte it reads, so it is usually limited by memory bandwidth, not by compute. See prefill vs decode on the roofline.
Scale and offset (asymmetric quantization): A stored integer q is turned back into a real number with x = s·q + z. One scale s and one offset z are shared by a whole group of values. See the quantization process.
Token-wise vs channel-wise group: A token-wise group shares one scale and offset across the channels of one token; a channel-wise group shares them across many tokens within one channel.
Tensor Core: The GPU unit that multiplies small matrix tiles in one instruction. It needs its inputs in a floating-point format such as BF16, FP8 or FP4, so packed integer codes must be turned into that operand format first. FOSCAR does this conversion with bit tricks but skips rebuilding each value with its scale and offset. See precision formats.
Instruction issue (lane-slot): Each GPU cycle, the warp schedulers can start only a fixed number of instructions. The paper counts this budget in lane-slots: one instruction issued for one lane of a warp.
Operand-fold / acc-fold: FOSCAR's two folding modes. Operand-fold multiplies the other input (the query or the attention weights) by the scale before the product; acc-fold multiplies the finished partial product by the scale. Both add a separate offset correction.
GQA: Grouped-query attention: several query heads share one KV head, which shrinks the cache. See GQA: shrinking the cache.

The news. On November 6, 2026, researchers from Together AI and the University of Sydney posted FOSCAR (arXiv 2611.09722), a kernel library for decode attention over 2-bit and 4-bit KV caches on NVIDIA H100 and B200. The paper reports that existing fused INT2 kernels reach under 31% (H100) and 15% (B200) of peak HBM throughput, traces the gap to per-value reconstruction work, and replaces that work with affine folding. The kernels are integrated into SGLang. Read the paper →

Picture a receipt in a foreign currency with three lines: 3 of an item stickered 2, 1 stickered 1, and 2 stickered 3. A slow cashier converts every sticker on its own (multiply by the exchange rate, add the per-item fee), then multiplies by the quantity and adds up the lines. A fast cashier first forms the weighted subtotal in raw stickers (3×2 + 1×1 + 2×3 = 13), multiplies that one number by the rate, and adds the fee times the total quantity (3 + 1 + 2 = 6). Both get the same total to the cent, because "multiply by a rate, add a fee" is an affine step, and a weighted sum lets you pull an affine step outside. If instead every line had its own rate, the fast cashier would multiply each quantity by its line's rate before adding: same idea, applied to the other side.

Now make the receipt 128 lines long, all sharing one rate and one fee. The slow cashier performs 128 conversions; the fast one performs two operations. That is the shape of a low-bit KV cache: each cached number is stored as a 2-bit or 4-bit code q, and a group of 128 codes shares one scale s and one offset z, so the real value is s·q + z (how that encoding works). Attention then multiplies those values by queries or attention weights and sums them, a weighted sum like the receipt. Affine folding is the fast cashier: build the Tensor Core inputs straight from the packed codes, let the matrix product form the weighted sum, and apply s and z once per group, either to the partial result or, when the scale changes along the sum, to the other operand.

Why does the slow cashier hurt so much on a GPU? Decode attention sits on the bandwidth-bound side of the roofline (prefill vs decode), so the time is mostly spent reading the cache from HBM (the GPU's main memory), and shrinking the cache from 16 bits to 2 bits should cut the bytes it reads by up to 8×. But every packed code still has to be shifted out of its byte, converted to a floating-point format, and rescaled before the Tensor Core can use it. That per-value instruction cost does not shrink when the bit width does, so at 2 bits the bottleneck moves from memory bandwidth to instruction issue. The trend makes it worse every generation: from H100 to B200, peak HBM throughput grew 2.39× while peak instruction-issue throughput grew only 11%, according to the paper.

Attention has two matrix products, and the fold depends on which axis each one sums over. QKᵀ sums over the head dimension (the channels of a key); PV sums over cached tokens. The rule is: if a group's scale is constant along the axis being summed, scale the finished partial sum (acc-fold); if the scale changes along that axis, pre-scale the other operand instead (operand-fold). In both cases the offset becomes one extra correction term, z times a sum of the other operand, added to the output. FOSCAR keeps the online softmax of FlashAttention and only adds ordering rules: key-side corrections must land before softmax, and value-side corrections before the final division by the softmax denominator. One limit is real: under acc-fold, an unscaled partial sum cannot cross a group boundary, so each group's partial product is scaled before it joins the running total.

Cache layout	Product and axis it sums over	Fold used	Where the scale goes
Token-wise K	`QKᵀ`, sums over channels	acc-fold	Multiplies each group's partial score; offset adds `z` × sum of the query
Channel-wise K	`QKᵀ`, sums over channels	operand-fold	Multiplies the query `Q` before the product
Token-wise V	`PV`, sums over tokens	operand-fold	Multiplies the attention weights `P` before the product
Channel-wise V	`PV`, sums over tokens	acc-fold	Multiplies each group's partial output; offset adds `z` × sum of the weights

Here is the instruction budget on an H100, using the paper's counts. Rebuilding four 2-bit values costs six instructions (two shifts and two logic ops to convert, two paired multiply-adds for s·q + z), and four 2-bit values are one byte, so reconstruction demands 6 lane-slots per byte. At peak HBM speed the H100 can issue about 9.99 lane-slots per byte, so reconstruction alone eats 60% of the issue budget before softmax or the matrix multiply get a single instruction. Folding changes the count of scale-and-offset operations: with 4 query heads per KV head and groups of 128, it needs 2 × 4 / 128 = 1/16 of an operation per stored value instead of 1, a 16× cut. Together with FOSCAR's bit tricks for building Tensor Core inputs, total INT2 demand on H100 drops to 4.25 lane-slots per byte. On B200 the gap is starker: reconstruction needs 8 lane-slots per byte against a budget of 4.65, and FOSCAR needs 1.00.

The measured payoff, on the paper's setup (32 query heads, 8 KV heads, head dimension 128, token-wise groups of 128): INT2 decode runs a geo-mean 3.36× faster than FlashInfer BF16 on H100 and 2.67× faster on B200, and INT4 runs 2.83× and 2.30× faster. Against B200's hardware-native NVFP4 path the margin narrows to 1.16× (INT2) and 1.00× (INT4), which is the honest comparison when the hardware already reads 4-bit floats natively. In SGLang, serving Qwen3-8B with 100K-token prompts and a 99% prefix-cache hit rate, swapping FOSCAR into OSCAR's INT2 setup raised peak aggregate throughput from 269 to 487 tokens/s on H100 and from 333 to 868 tokens/s on B200.

Folding makes 2-bit decode fast; it does not make 2-bit decode accurate. The fold is exact in arithmetic: measured against the dequantized INT2 cache, FOSCAR's kernel error is about 1.7 × 10⁻³, the same as BF16 decode. The much larger error of the INT2 output on the paper's stress test (an L2 distance from the exact output of about 0.5–0.6) comes from 2-bit quantization itself, so a real deployment still needs an accuracy recipe, such as OSCAR's rotations and full-precision windows of recent tokens. Folding also has a break-even point: it costs 2·h_q / g operations per value, so it only wins while the group size g is larger than twice the number of query heads per KV head. With GQA ratios of a few heads and groups of 32 or more, that holds comfortably. The same algebra appears for weights in the fused INT8 GEMM epilogue; FOSCAR's contribution is doing it inside attention, where two different reductions, online softmax and per-group boundaries all constrain where the scale may go, a cousin of operator fusion.

Goes deeper in: LLM Internals → Quantization → The Quantization Process

Related explainers

KVarN — Hadamard-rotated 2-bit KV cache — the accuracy side of 2-bit KV: how to make the codes good. FOSCAR is the speed side: how to read them fast.
D-Quant — entropy-coded KV cache — another KV format that has to keep a regular layout so attention kernels can decode it quickly.
Fused INT8 GEMM on Tensor Cores — the same "apply the scale after the multiply" idea, for weight matrices.

Continue in trackQuantization: how a scale and offset turn small integers back into real numbers

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based