BitSparse is a GPU format that **losslessly compresses the activation cache** of ReLU and squared-ReLU feed-forward layers during training. It stores a bitmask of nonzero positions (one bit per tile position), the packed nonzero values and a per-tile prefix count.

Why not just use activation checkpointing?

Checkpointing saves memory by **recomputing activations**, which adds forward compute. BitSparse keeps the exact values and avoids recomputation, so it sits at a different point on the memory-versus-compute trade-off. In the paper's LLM runs checkpointing uses less memory but is slower.

Does BitSparse change the training result?

No. It is lossless, so the cache is recovered exactly. The paper reports gradients equal up to floating-point rounding and a final validation loss virtually identical to the original Modded-NanoGPT recipe.

When does BitSparse not help?

It needs **exact zeros** in the activations, so it relies on ReLU-family activations. It covers feed-forward activations only, not attention or Mamba layers, and gated activations such as SwiGLU are future work.

GPU· 2026-10-17

BitSparse — Lossless sparse activation cache — What does it mean?

TL;DR

What is it: BitSparse (arXiv 2610.22256, October 2026) is a GPU format that stores the training-time activation cache of ReLU and squared-ReLU feed-forward layers as a bitmask plus only the nonzero values, which is the concept this article explains.
Why it’s needed: Saved activations grow with every token in the batch, so they limit batch size and sequence length even after the weights fit; the paper reports about 30% lower VRAM per token on two models, which can leave room for longer sequences or larger batches, without changing the math.
vs previous: Unlike activation checkpointing, which throws the cache away and recomputes it (extra forward compute), and unlike lossy compression, which changes the stored values, BitSparse keeps every value exactly and skips recomputation by exploiting the exact zeros ReLU already produces.

Jargon

Activation cache: The intermediate values a layer computed in the forward pass and keeps in GPU memory because the backward pass needs them to compute gradients.
ReLU and squared ReLU: Activation functions that turn every negative input into an exact zero; squared ReLU then squares the positive values. Both leave many exact zeros, which is the sparsity BitSparse uses.
Activation sparsity: The fraction of activation entries that are exactly zero. Higher sparsity means more empty hangers and more savings.
Activation checkpointing: Saving memory by not storing some activations and recomputing them during the backward pass, which costs extra compute. See the memory ladder.
Lossless vs lossy compression: Lossless compression restores the original bits exactly; lossy compression restores an approximation. BitSparse is lossless.
Bitmask: An array with one bit per tile position, marking which positions hold a nonzero value.
Prefix array: A running count of nonzero values per tile, so a reader can jump straight to the packed values of any tile.
Host synchronization: The CPU waiting for the GPU to finish and report a result (here, how many nonzeros there are) before it can allocate memory. It stalls the GPU's job queue.
BF16 and FP8: Two low-precision number formats that use 16 and 8 bits per value. Each stores a sign bit, which BitSparse can drop for nonnegative activations.

The news. On October 15, 2026, a Cambridge and Oxford team posted BitSparse, a lossless way to store ReLU-family feed-forward activations during training. The authors report 31% lower VRAM per token on Modded-NanoGPT and 30% on Nemotron-H, with overhead of no more than 1% at practical sequence lengths. Read the paper →

Picture the training step as a coat-check room. Every forward pass hangs a coat on a hanger for each activation, and each coat must stay on the rail until the backward pass reaches that layer. The rail grows with every token in the batch, so a bigger batch means a longer rail, and the rail is what runs the GPU out of memory first. See the HBM step of the GPU memory hierarchy for why that memory is the scarce resource.

Now look at the rail. Squared ReLU (used in the Nemotron family and Modded-NanoGPT) turns every negative input into an exact zero, so most hangers are empty but still reserve their slot. The paper's own illustration shows a row where 93% of entries are zero. A dense cache pays for all of them.

BitSparse keeps a short list instead: one tick box per hanger, plus only the coats that exist. Because the tick boxes cost one bit each and a BF16 coat costs 16, the room shrinks by a large factor wherever hangers are mostly empty. And the backward pass loses nothing, since the gradient needs only the positive values and their positions.

Here is the format. The tensor is cut into tiles, and each tile stores three things: a bitmask (512 bytes per tile), a packed array of its nonzero values, and a 32-bit prefix count telling a reader where that tile's values start. The prefix is the number on the rack: with it, every tile can be decoded in parallel, with no scan through earlier tiles.

Two extra tricks matter. First, a squared-ReLU or ReLU output is never negative, so the sign bit of every stored value is redundant; the optional sign-bit packing drops it, so BF16 values take 15 bits and FP8 values take 7. Second, the number of nonzeros is only known on the GPU, and asking the CPU to allocate exactly that much would force host synchronization. BitSparse instead appends every layer's values to one large preallocated buffer and records offsets on the device, which keeps the queue full and works with torch.compile. Compression and decoding are custom Triton kernels fused into an autograd function (see how Triton works and operator fusion).

Approach	What it keeps for the backward pass	Exact?	What it costs
Dense cache	Every activation value	Yes	Most memory
Activation checkpointing	Nothing; recomputes the forward pass	Yes	Extra compute: in the paper's MLP batch-size sweep a checkpointed step took 4,268 ms versus 3,604 ms for BitSparse at 32,000 rows (Table 4)
Lossy compression (for example random projections)	An approximation of the values	No	Changed gradients
BitSparse	Bitmask, nonzero values, per-tile prefix	Yes	Needs ReLU-family zeros; covers feed-forward layers only

Now the worked example, using the paper's format with numbers I chose (illustrative). Take one tile of 4,096 BF16 activations at 90% zeros. Dense storage costs 4,096 × 2 bytes = 8,192 bytes. BitSparse stores a 512-byte bitmask, a 4-byte prefix, and about 410 nonzero values × 2 bytes = 820 bytes, about 1,336 bytes in total, roughly a 6.1× smaller cache for that tile. Sign-bit packing trims the values to about 769 bytes, giving about 1,285 bytes (roughly 6.4×). At 50% zeros the same tile costs about 4,612 bytes, only 1.8× smaller, because the fixed bitmask now matters. Working it out from the format, the break-even is near 6% zeros for BF16: below that, the bitmask costs more than the empty slots save.

Why then is the measured saving only about 30% per token, not 6×? Only the feed-forward activation cache shrinks. Attention, Mamba layers and other tensors are untouched. The paper measures 0.402 MiB per token falling to 0.277 MiB on Modded-NanoGPT, and 2.719 MiB to 1.894 MiB on Nemotron-H.

Two honest limits. The gain depends on exact zeros, so it follows the fate of ReLU-family activations; gated activations such as SwiGLU are left to future work. And checkpointing still uses less memory in the LLM runs, but at a notable runtime cost; BitSparse sits at a different point on the memory-versus-compute curve, with decoding reaching 97.4% of attainable memory bandwidth in the paper's benchmark. In a full Modded-NanoGPT training run, the paper reports a validation loss virtually identical to the original recipe, and a larger microbatch that ran out of memory with standard storage fit with BitSparse.

Goes deeper in: GPU & CUDA → Memory Hierarchy → HBM: Where Your Model Lives

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based