What is selective precision streaming?

It is the technique at the core of **StrataFlux**: keep compact low-bit weights and KV cache in GPU memory, and fetch extra refinement bits from host RAM or an SSD only for the weight channels with the largest activations and the cached tokens with the most attention at each decode step. Fetches start early and are reused across steps, so most of them do not stall generation.

Why not fetch full-precision weights from host memory instead?

Host memory is reached over a link that is far slower than GPU memory, so fetching full-precision data makes generation wait. StrataFlux fetches only the extra bits that refine values the GPU already holds, and only for a small selected share; at 32 channels per 1,024 with 4-bit refinements, that is about 0.125 extra bits per weight per step, against 16 bits for a BF16 copy.

How does it relate to KV cache offloading and sparse attention?

Offloading keeps full-precision data in a slower tier and brings it back when needed; sparse attention computes attention over a selected subset of tokens only. StrataFlux keeps every token in attention at low precision and sharpens only the selected ones with fetched bits, and the paper reports lower KL divergence than both sparse attention and full 16-bit replacement at matched bit budgets.

Does it need a special quantizer?

Weight refinement keeps the resident quantizer's format and kernels, and the paper tests it on AWQ, YAQA, LNQ and SqueezeLLM bases. KV refinement is co-designed with the KVarN cache format, with adaptations to other formats described in an appendix.

LLM· 2026-11-10

StrataFlux — Selective precision streaming — What does it mean?

TL;DR

What is it: StrataFlux (arXiv 2611.08363), from Cornell University and Carnegie Mellon University, is an LLM inference system built on selective precision streaming: low-bit weights and KV cache stay in GPU memory, and extra precision bits for the few weight channels and tokens that matter at this step are fetched from host RAM or an SSD.
Why it’s needed: Low-bit quantization is what lets a model and a long context fit in GPU memory and decode fast, because decode is memory-bandwidth-bound, but 2-bit codes lose accuracy; StrataFlux recovers much of that accuracy without giving back the memory savings and while keeping most of the speed.
vs previous: Before, the choice was quantization, which throws the lost precision away, or offloading, which keeps full precision in a slower tier and can make generation wait for it; StrataFlux uses both tiers at once, fetching only refinement bits for selected data, started before the step that needs them.

Jargon

Quantization: Storing each number with fewer bits, such as 2 or 4 instead of 16, by snapping it to one of a few allowed levels. See How Numbers Shrink.
KV cache: The keys and values of every earlier token, kept in GPU memory so each new token does not recompute them. It grows with context length; see Memory Cost.
Host RAM and SSD: The CPU's main memory and the solid-state drive. Both are much larger and cheaper than GPU memory, and much slower for the GPU to read from. On a unified-memory machine, where the CPU and GPU already share one RAM pool, StrataFlux keeps refinement bits on the SSD and caches recent ones in RAM.
Residual: The error a quantized value leaves behind: the original value minus its low-bit approximation. Refinement bits encode part of it.
Nested quantization: A code layout where extra bits pick a finer point inside the bin the low-bit code already names, so the low bits stay valid on their own and the extra bits only sharpen them.
Projection and channel: A projection is one weight-matrix multiply inside a layer; a channel is one input dimension of it. A channel with a large activation multiplies its weight column's error by a large number.
Fetch lead and fetch lag: Starting a precision fetch one or more layers before the projection that needs it, or adding the correction one or more layers after, so the transfer overlaps other work.
Perplexity and KL divergence: Two measures of how far a model's predictions are from a reference model; lower is better for both.

The news. On November 5, 2026, researchers from Cornell University and Carnegie Mellon University posted StrataFlux (arXiv 2611.08363; the paper calls the name provisional). Implemented in llama.cpp with CUDA and Metal backends, it reports that for 2-bit Llama-3.1-8B-Instruct weights, WikiText-2 perplexity falls from 9.46 to 8.40 while keeping 95.4% of the quantized model's throughput on an RTX 5090, and that with 2-bit weights and a 2-bit KV cache, Qwen3-8B's AIME 2024/2025 accuracy rises from 11.0% to 47.1% at unchanged GPU memory use. Read the paper →

Think about how a map app stays fast on a phone. It never downloads the whole world at full detail; it keeps a blurry base map everywhere and streams sharp tiles only for the part you are looking at. If you pan faster than the tiles arrive, you see the blurry version for a moment, and nothing breaks. StrataFlux treats a language model's numbers the same way. The resident base is the blurry map: weights quantized to 2–4 bits and a KV cache at 2–3 bits, held in GPU memory, where the GPU reads them at HBM bandwidth. The sharp tiles are refinement bits, kept in host RAM on a machine with a dedicated GPU, or on an SSD on a unified-memory machine such as DGX Spark, where the CPU and GPU already share one pool of RAM.

Which tiles count as "where you are looking" changes on every token. For weights, a projection's error is the sum of each input channel's activation times that channel's weight error, so a channel with a large activation magnifies its own rounding error, the same reason outlier channels hurt quantization. Following an earlier method, DecDEC, StrataFlux splits each projection's input channels into chunks of 1,024 and refines the k channels with the largest activation magnitude in each chunk at this step. For the KV cache, it scores every cached token by the attention it receives, computed from the resident low-bit keys, and refines the highest-scoring tokens for each KV head, the set of keys and values that a group of query heads shares. Everything not selected still takes part in the computation at low precision; it is just not sharpened.

32-bit float — virtually continuous

What travels over the link is not a full 16-bit copy. The fetched bits subdivide the bin that the resident code already names, so a few extra bits buy a finer value without resending the coarse part. This is nested quantization, a principle StrataFlux adapts from Any-Precision LLM, where it was used for weights, to the KV cache. A 2-bit key can land in one of 4 bins; 2 fetched bits split each bin into 4 equal parts, giving the selected token the resolution of a 4-bit code. An earlier system, SpeCache, also kept a full low-bit cache but fetched complete 16-bit replacements for the selected tokens. In the paper's measurement at matched key-plus-value bit budgets, nested refinement with 4 or 8 extra bits per element gives lower KL divergence than both that approach and sparse attention, which leaves the unselected tokens out of attention altogether. For weights, the refinement values are also fitted, not just copied: they are adjusted on a small calibration set so the selected columns partly make up for the error of the channels left out.

The map app also loads tiles before you pan to them. A refinement fetched only once a projection's input is ready, and needed before its output is used, has a small window to hide in. StrataFlux widens it: fetch lead picks the channels and starts the transfer one or more layers early, from an earlier activation, and fetch lag adds the correction one or more layers later, so the transfer overlaps the projections in between. In the paper's Nsight Systems trace on Llama-3.1-8B-Instruct (3-bit AWQ, k = 32, RTX 5090), fetch lead and lag at a distance of one block cut the shown block's execution time by 30.7% against the projection-local schedule. Selections also repeat from one token to the next, so on SSD-backed machines fetched bits are kept in a RAM cache and reused across steps. On a miss, the runtime can either wait or use the low-bit value now while a CPU thread loads the bits for later steps; the paper reports that skipping gave the better latency–accuracy tradeoff at the small slowdown budgets of its unified-memory results, the blurry tile rather than a frozen screen.

Approach	What stays on the GPU	What crosses the link	What unselected data gets	Source
Static low-bit quantization	low-bit weights and KV	nothing	the low-bit value; the lost precision is gone	§2.1
Offloading	part of the data	full-precision data, which generation may wait for	full precision, once it arrives	§1
Sparse attention (top-k, Quest)	selected tokens	the selected 16-bit entries	left out of attention	§2.3
SpeCache (16-bit fetching)	a full low-bit KV cache	complete 16-bit entries for selected tokens	the low-bit value	§2.2
StrataFlux	low-bit weights and KV, plus a refinement cache on SSD machines	only the refinement bits for selected channels and tokens, started early	the low-bit value	§3

A worked example, combining two of the paper's settings (illustrative arithmetic, not a measured configuration). Take one weight projection with a budget of k = 32 channels per chunk of 1,024, the setting in the paper's fetch-lead and fetch-lag sweep, and 4-bit refinement values, the encoding StrataFlux uses for weight refinements. Per decode step, the share of columns fetched is 32 ÷ 1,024 = 3.1%, and each fetched weight carries 4 bits, so the link carries on average 0.031 × 4 = 0.125 extra bits per weight, before scale factors and index metadata. The GPU also reads the resident 3-bit weights at every step. So the backing tier adds about 4% to the bits per weight the step touches (3 + 0.125 = 3.125), while sending a BF16 copy of every weight over the same link would move 16 bits per weight, 128 times the refinement payload. The real selection changes every step and adds metadata, but the ratio shows why streaming refinements over PCIe is affordable when streaming whole weights is not.

Where it earns its keep: GPU memory is the limit, and the machine has spare RAM or SSD. The paper evaluates desktop and workstation GPUs (RTX 5090, RTX 5070 Ti, RTX PRO 6000) and unified-memory machines (DGX Spark, MacBook Air M3). At 128K context, Llama-3.1-8B-Instruct with 3-bit weights and a 2-bit KV cache reaches PG-19 perplexity and RTX 5090 throughput comparable to vLLM running 4-bit AWQ weights (a widely used weight-quantization method) with an 8-bit KV cache, using approximately half the GPU memory. The overlap relies on spare GPU cores: refinement runs beside projections that are memory-bandwidth-bound during decode, so the compute units would otherwise sit partly idle. Settings are tuned per model, quantizer and platform under a user-set slowdown limit, and KV refinement adds a step that cannot overlap the surrounding attention, because which tokens to refine depends on the attention scores themselves.

Goes deeper in: GPU & CUDA → Memory Hierarchy → NVLink & PCIe

Related explainers

BitNest: bit-nested draft and target weights — the same base-plus-residual bit split, used to share one set of weights between a speculative drafter and its target
FlashOffloading: full host-DRAM KV offload — the offloading side of the tradeoff, moving whole KV entries with a hot cache on the GPU
FOSCAR: affine folding of low-bit KV dequantization — making the resident 2-bit and 4-bit KV cache cheaper to compute with

Continue in trackMemory Hierarchy: why PCIe is the slow rung, and what crossing it costs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based