DictLLM compresses LLaMA-7B 5.1× — Shared dictionary plus sparse coefficients — What does it mean?
The news. On November 5, 2026, Jinho Park, Se Young Chun and Mingoo Seok posted DictLLM to arXiv. It compresses LLaMA-7B by 5.1× with a 2.9-point rise in WikiText-2 perplexity, which the authors report as 3.6× higher compression than the best low-rank method they compare against. Read the paper →
Picture a restaurant chain with several branches. The usual way to cut the chain's storage bill is to give every branch a smaller private pantry. That is per-layer low-rank compression, and it fails when the cuts get deep, because a tiny pantry cannot cook the whole menu. DictLLM does what a real chain would do: it stocks one central pantry of ingredient jars for a group of branches, and each branch keeps only a short order slip naming the few jars its dishes need and how much of each. Every dish is still cooked. Only the shelf space is shared. To save paper, the jar labels and the slips are then printed in small NVFP4 (4-bit) type.
Underneath the metaphor, a branch is one layer's weight matrix, for example the query projection, and the jars are the columns of a dictionary matrix D. A layer's weights are rebuilt as D times a coefficient matrix C, written W ≈ D · C. D is shared by every layer of the same kind in a group, and only C is specific to a layer. C is also made sparse, so most of its entries cost nothing to store. The idea comes from classical dictionary learning (the Method of Optimal Directions and k-SVD), which rebuilds images from an overcomplete dictionary and a few nonzero coefficients. DictLLM applies it to the weights of a pretrained LLM.
DictLLM runs four steps on the pretrained model, using calibration text (256 WikiText-2 samples of 2,048 tokens) and no further training:
- Factorize. Scale each weight matrix by the statistics of the inputs it sees, take an SVD of the group's matrices, and keep the top directions as the shared dictionary D. What is left becomes each layer's coefficients C.
- Prune the coefficients. Zero out the least important entries of every C until the size target is met. Importance comes from a dictionary-aware Hessian, which scores how much a layer's output changes if one coefficient is removed. Because this works in coefficient space, a single matrix inverse is shared across all coefficient columns.
- Refit the dictionary. D was computed while C was still dense, so after pruning it is adjusted to fit the sparse C.
- Quantize both factors to NVFP4, using the input covariance for D and the dictionary-aware Hessian for C.
| Approach | What it shrinks | Limit the paper points to |
|---|---|---|
| Low-rank (ASVD, SVD-LLM) | each matrix becomes two thin matrices | needs a retained rank far below the matrix size; deep cuts lose spectral components that still matter |
| Pruning (SparseGPT) | individual weights, matrix shape kept | gains depend on sparse-matrix support in the hardware |
| Quantization (GPTQ, AWQ) | bits per number | the parameter count stays the same |
| Parameter sharing | weights reused across layers | usually trained from scratch, so hard to apply to a finished model |
| DictLLM | shared dictionary, sparse coefficients, then 4-bit storage | the sparsity is unstructured, so speed depends on irregular-access kernel support |
Why shrink weights at all? Decoding a token reads nearly every weight once, so on memory-limited hardware the speed is set by bytes moved, not by arithmetic. That is the memory-bound regime of the roofline model, and the authors, who target phones and similar devices, say the main gain of compression comes from reduced memory traffic rather than fewer multiplications. Fewer stored bytes means fewer bytes to read.
Take an illustrative group of 4 layers, each a 4,096 × 4,096 matrix (illustrative, not the paper's setting). Dense, that is 4 × 16.8M = 67.1M parameters. Factor them with a shared dictionary of rank 2,048: D is 4,096 × 2,048 = 8.4M parameters, stored once, and each layer's C is 2,048 × 4,096 = 8.4M. Prune half of every C and each layer keeps 4.2M, so the group costs 8.4M + 4 × 4.2M = 25.2M parameters, 2.7× fewer, before counting the index bits that mark which entries survive. Spend the same 25.2M on four private low-rank factorizations (two 4,096 × r matrices per layer) and r can only be 768, while the shared dictionary lets every layer choose from 2,048 directions. Storing everything in roughly 4 bits instead of 16 then stacks up to about another 4×, less the cost of the scale factors.
The measured gains are real but smaller than the headline 5.1×, which counts all four steps. In the paper's decoding benchmark, a LLaMA-7B model with the same factorization rate and about 44% coefficient sparsity ran at 126.81 tokens/s, 1.24× the original model and 1.12× the factorization-only model, and its weight memory fell from 13.48 GB to 7.82 GB (1.72×), on the tested hardware and sparse backend. At 2× compression, its WikiText-2 perplexity is 9.4 points lower than Basis Sharing's. The limits are the authors' own: pruning leaves unstructured sparsity, so the speed-up needs kernels that handle irregular access, and they name block-structured sparsity as future work. The headline figures are for LLaMA-7B and perplexity; the paper also reports zero-shot accuracy and other model families.
Goes deeper in: LLM Internals → Quantization → Modern Methods
Related explainers
- SigmaScale: learned SVD scaling — the per-matrix SVD route that DictLLM extends across layers
- iS-KV: basis-synchronized low-rank KV compression — a shared low-rank basis again, applied to the KV cache instead of the weights
- E8-QAT: 2-bit training with an E8 lattice codebook — the other axis, fewer bits per weight instead of fewer weights