LILA prunes LLM neurons without calibration data — Calibration-free structured neuron pruning — What does it mean?

The news. On September 10, 2026, a six-author paper introduced LILA — Latent-Informed Layer Analysis — a way to choose which feed-forward neurons a large language model can lose by reading the weight matrix and nothing else. Existing structured-pruning methods all want something extra at pruning time: SliceGPT calibrates on WikiText-2 text, PruneNet trains a 45M-parameter reinforcement-learning policy. LILA wants none of it, reports 1.57 pp more zero-shot accuracy — accuracy on benchmark tasks the model was never fine-tuned for — than PruneNet on LLaMA-2-7B at 25% sparsity with no fine-tuning, and preserves the original architecture. Read the paper →

Picture a city skyline seen from across the river, and imagine the buildings lined up tallest first. That profile is the city's signature: a few towers on the left, a long tail of low blocks on the right. An FFN weight matrix has exactly this kind of profile. Its singular values — the sorted strengths of its internal directions — are the building heights, and the shape of that sorted list says how much genuinely distinct work the matrix is doing.

Now demolish one building and redraw the silhouette. If the city had a dozen blocks of about that height, the neighbours close the gap and the profile barely moves. If you took out the one tower, a notch opens that nothing else fills. LILA's bet is that a neuron is safe to delete when the skyline closes back up without it. That sentence is the whole scoring rule, and the picture above is the measurement: green is the profile before, pink dashed is the profile after, and the amber caliper marks the widest gap between them.

What makes this worth a paper is the survey you no longer run. The usual way to find the redundant blocks is a traffic study: send real sample text through the model and let what lights up decide what can go. That study needs a corpus, a forward pass over it, and a judgement about whether your corpus resembles the traffic the model will actually see. LILA skips it and reads the buildings.

Concretely: take one FFN weight matrix, run a singular value decomposition, and keep the sorted list of singular values. Now zero out neuron i — a LLaMA-style FFN holds three weight matrices per layer, called gate, up and down, and one neuron owns a row of the gate, a row of the up, and a column of the down — and decompose again. You now have two distributions of singular values, and the Kolmogorov-Smirnov distance between them is a single number: the widest vertical gap between their cumulative curves. Small gap, redundant neuron. Large gap, load-bearing neuron. LILA replaces the whole calibration step with one closed-form comparison between two singular-value distributions.

The paper backs the intuition with a Neural Tangent Kernel analysis — NTK being a standard way to approximate how much a network's function moves when its weights move — and reports a 22× reduction in functional distortion versus removing neurons at random. That is the claim that matters: the spectrum is not just a convenient thing to measure, it tracks the damage. Which parts of a model a compression method is allowed to touch is the same question the curriculum asks under what to quantize; LILA answers it for whole neurons rather than for bits.

Hold two things fixed — the model is LLaMA-2-7B and the setting is the paper's 25% sparsity — and the demolition adds up quickly. LLaMA-2-7B's published configuration is 32 layers with a hidden width of 4096 and an FFN width of 11008, and each FFN neuron owns three slices of weight: a row of the gate projection, a row of the up projection, and a column of the down projection. So one neuron costs 3 × 4096 = 12,288 weights. A quarter of 11008 is 2,752 neurons per layer, which is 2,752 × 12,288 ≈ 33.8M parameters per layer, and across 32 layers that is ~1.08B parameters — roughly 2.2 GB at fp16, taking the model from about 13.5 GB down to about 11.3 GB. (That arithmetic is ours from the published shapes, not a figure the paper quotes.) The part worth sitting with is the price of the decision: those 2,752 neurons per layer were chosen with zero forward passes and zero calibration tokens — the cost is spectral comparisons between the full and the neuron-ablated weight matrices, never a pass over a corpus. PruneNet needed a 45M-parameter policy trained before it could pick a single neuron.

The diagram above is the site's map of quantization sensitivity, and it is here as a comparison rather than as evidence about LILA. What it shows is that a transformer does not tolerate compression evenly down its depth — which is exactly the question a pruning budget has to answer too. LILA's answer is to extend the KS score from ranking neurons to allocating a per-layer budget, which the paper reports yields state-of-the-art generative preservation at moderate compression. The KS score tells you where the budget can go, and at higher compression the paper reports single-layer architectural bottlenecks that limit how far it can go.

Two honest limits. The reported numbers cover LLaMA-2-7B and Phi-2 at the stated sparsities, so this is evidence about a shape of model rather than a law about all of them, and the headline comparison is against methods that were themselves handed a calibration set — a fair fight for the claim being made, not proof that calibration never helps. What the result does establish is narrower and still useful: for FFN neurons, much of what a calibration pass was telling you was already sitting in the weight matrix's spectrum, free to read.

It also lands next to the other way of making a matrix smaller. Quantization keeps every neuron and shrinks the numbers; structured pruning keeps the number format and removes neurons. They compose, and the reason both live in the same corner of the curriculum is that they spend the same budget — memory bandwidth — in different currencies.

Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network

Related explainers

• SigmaScale learns its SVD scaling matrices — the other way to use a weight matrix's spectrum: shrink it by rank instead of deleting neurons.
• MAESTRO prunes MoE experts with a Markov chain — the same delete-whole-units idea one level up, at the expert rather than the neuron.
• KVARN's Hadamard 2-bit KV cache — another calibration-free compression result, applied to the KV cache rather than the weights.