Co-design FP–INT GEMM hardware for 2–3-bit LLM weights — Why low-bit weights save memory but not compute — What does it mean?

The news. On October 21, 2026, a UC Irvine team posted FIGFLEX to arXiv: a mixed-precision FP–INT GEMM accelerator co-designed with a 2–3-bit weight quantizer. On LLaMA-2-7B it reports WikiText-2 perplexity of 5.85 at 3-bit and 8.77 at 2-bit average precision. Read the paper →

Picture the cashier from the card above. The store has switched to vacuum-sealed packs, so the warehouse is smaller and trucks make fewer trips. But the only scanner at the counter reads full-size items, so every pack is cut open before it is scanned, and the scan costs the same whether the pack was sealed tight or loose. That is what the dequantize-to-FP16 kernels do with low-bit weights. Kernels such as CUTLASS mixed-input, Marlin and FLUTE load the small integer, convert it to FP16, and issue an ordinary tensor-core multiply. The paper's point is that fewer bits save memory traffic but not arithmetic, because the multiplier never sees the narrow number.

Whether that matters depends on where the workload sits on the roofline. When decode is memory-bound, moving fewer bytes is the whole win. When the multiply is the limit, as in prefill or decode at moderate batch size, the paper says the gain from these kernels saturates, since the arithmetic cost does not fall with the bit-width. Existing accelerators only partly fix this: bit-parallel units such as FIGNA and AxCore process every weight at one fixed width, and the bit-serial FIGLUT does scale with precision but spends its area on a lookup table that grows exponentially with the group size.

FIGFLEX's first idea is to stay in the integer domain. Activations are aligned to a shared exponent so they become integer significands, and then the dot product is a pure integer reduction, converted back to floating point once at the end. The second idea is radix-4 recoding: group the weight bits in pairs, so a 4-bit weight becomes two base-4 digits and each digit adds one partial product instead of one per bit. That roughly halves the partial products, and the recoding itself is exact, so it adds no arithmetic error of its own (the quantization error is a separate matter). In the cashier's terms, the scanner reads two bits of the pack per beep.

A wrinkle appears at 3 bits. Two bits fill a digit, so a 3-bit weight leaves a half-empty lane. FIGFLEX's INT3 trick, called high-bit lane fusion, packs the leftover high bit of two weights into one shared lane: two 3-bit weights then need three lanes instead of four. A small weight aligner at the array edge reorders the bits to feed this lane, and because it only reorders bits, it is exact too. The table shows the resulting throughput of one processing element.

The hardware only helps if the model really contains 2–3-bit weights without losing quality, so the second half of the paper is the quantizer. It assigns each group of channels more or fewer bits using the error that GPTQ could not compensate, rather than a salience score computed before quantization. The reason is that GPTQ already moves the rounding error of important columns onto less important ones, so a pre-quantization score promotes groups that would have been fixed anyway. With column scaling on, the paper reports that salience-based allocation is no better than uniform 3-bit on Llama-3.1-8B (7.237 against 7.227 perplexity).

The quantizer also divides each input channel by its own scale, which flattens outlier columns (the outlier problem). Without it, the paper reports 2-bit perplexity on Llama-3.1-8B rising from 13.77 to 16.48. The scale is rounded to a power of two so the scaled activations stay out of the FP16 subnormal range that the hardware's input stage flushes to zero, which, the paper says, lets the quantized model run on FIGFLEX at the same accuracy as on a GPU.

Here is the arithmetic for one processing element, with the model size chosen for illustration. Take a 7-billion-weight model (illustrative): stored in FP16 it needs 14 GB, and at a 3-bit average it needs 7 × 10⁹ × 3 / 8 ≈ 2.6 GB, about 5.3× smaller. On a GPU the multiply count does not change: one FP16 MAC per weight in both cases. Now take one 4096 × 4096 layer applied to one token, which is 16,777,216 weight-MACs. At INT4 speed (8 MAC per cycle) one FIGFLEX element needs 2,097,152 cycles; at INT3 (10.67) it needs 1,572,864; at INT2 (16) it needs 1,048,576. Going from 4 bits to 2 bits halves the cycles, which is the compute saving that dequantize-to-FP16 kernels cannot show. The paper's models use mixed widths per channel group, so these rows give the per-element rate, not a prediction of a whole model's speed.

What the paper shows, and what it does not: on a 3-bit mixed-precision model it reports whole-chip throughput per area that is 1.56× that of AxCore and 1.42× that of FIGLUT on the linear GEMMs. Those are accelerator-design comparisons, not measurements on a shipping GPU, and the paper does not claim a speedup for existing GPUs. The durable lesson is the co-design: the bit-width a quantizer picks only pays off in compute if the datapath can scale with it.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Precision-Throughput Tradeoff

Related explainers

• INT8 finally beats FP8 on consumer GPUs — Fused INT8 GEMM kernel — another case where integer tensor-core arithmetic, not storage, decides the speed.
• N-Gram Memories at Two Bits — Row-addressable lookup-table quantization — what going down to 2 bits looks like on the memory side.