Your Transformer Is Secretly Elastic — Joint neuron-head ranking — What does it mean?

The news. On October 14, 2026, researchers at the University of Amsterdam and TNO posted Your Transformer Is Secretly Elastic to arXiv. The paper reports that one jointly optimized ranking lets 27 pretrained vision and text encoders, from 22M to 8B parameters, degrade smoothly up to 60% sparsity, with no retraining and no weight correction. Read the paper →

Picture one packing list for trips of any length. A model family usually ships a few fixed bag sizes, small, base and large, and pruning a model to an in-between size produces yet another fixed bag. This paper keeps one bag and writes down one ranked list of everything that could go in it. For a short trip you pack from the top of the list and stop early. For a longer trip you keep packing. Every budget is just a different place to stop reading the same list.

The items on the list are small. Inside each Transformer block they are the individual MLP neurons of the feed-forward network and the individual attention heads. A block with 3,072 hidden neurons contributes 3,072 items, not one. The hard part is deciding the order, because a neuron in block 2 and a head in block 9 have to be compared on the same scale.

The paper builds the order in two stages. First it gives every unit a starting score, the empirical Fisher: the average squared gradient of the loss over a small calibration set. The authors call these scores a weak basis for pruning for three reasons: the diagonal approximation ignores how units interact, the proxy loss may not match downstream performance, and scores are not comparable across blocks or unit types. So the second stage repairs the ranking by gradient descent while the model weights stay frozen. Each score becomes a learnable log-score, centered by its median per unit type so neurons and heads start from the same reference point.

The problem is that choosing the top units is a hard cut, and a hard cut has no gradient. The paper replaces it with a soft mask. For a sparsity target b, it walks the current ranking from the top, keeps units until their cumulative FLOP cost reaches the allowed share, and records the score of the last unit kept as the cutoff. Each unit's mask is a sigmoid of its distance from that cutoff, so units far above it get a mask near 1 and units far below get a mask near 0. The loss under that soft mask is differentiable with respect to the scores. The method runs the same calibration batch under the masks for many targets, averages the losses, and updates only the log-scores.

Using many budgets at once matters because mask gradients concentrate near each cutoff. A single budget only teaches the part of the list around its own cutoff. Several budgets together expose different regions of the same list. The paper reports that this jointly optimized ranking outperforms separate rankings optimized for individual targets.

Here is why the per-unit score matters, using the prior method's limitation. That earlier search adjusted one scale per block, so the order of neurons inside a block never changed. If neuron 3,000 in a block is important for the task but its starting score is low, keeping it means keeping the 2,999 neurons ranked above it. Per-unit scores remove that coupling: the neuron can move up the list on its own, and the budget only has to cover the cost of that one neuron.

At runtime nothing is optimized. A budget is chosen, the ranking is hard-pruned at that cutoff, and the smaller model runs. The paper reports construction in minutes, with time scaling sub-linearly in parameter count.

A worked example with named numbers (illustrative). Take a ViT-B/16-shaped model with 12 blocks, 3,072 MLP neurons and 12 heads per block. That is 12 × (3,072 + 12) = 37,008 ranked units, so the whole elastic model is the original weights plus 37,008 scores. Serving five load levels the old way means five stored pruned models; here it means one model and one list.

Now the payoff. For a ViT-B/16 pruned to 60% FLOP sparsity, the paper reports up to 1.80× higher throughput and 1.26× lower single-query latency on an A100. If the dense model served 1,000 images per second at 10.0 ms per query (illustrative), the pruned one would serve about 1,800 images per second at about 7.9 ms. A 60% FLOP cut would be an ideal 1 ÷ 0.4 = 2.5× speedup, so the measured 1.80× realizes 72% of the ideal. The paper does not attribute the remaining gap here, but it devotes an appendix section to whether pruning translates to speedups.

Read the scope carefully. All 27 evaluated models are encoders for vision and text, including Qwen3 4B and 8B and E5-Mistral 7B used as text-retrieval encoders. The paper reports no results for autoregressive generation, so how well this ranking carries over to a chat model decoding token by token is not documented there. Deep pruning also stays costly on some backbones: in the fixed-dense-index test at 50% sparsity, DeiT keeps 66.3% k-NN accuracy while SigLIPv2 keeps 13.9%. The headline number, a 0.8-point accuracy drop when pruning DINOv3 ViT-H+/16 by 50%, which removes 427M parameters, is one backbone's result, not a guarantee for every model.

Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network

Related explainers

• Save 25% training FLOPs with tuned layer dropout — makes whole blocks optional, the depth-wise counterpart to ranking individual units.
• Compress context up to 266× with one ratio-adaptive model — another single model whose prefixes serve many budgets, applied to context compression.