What is contraction-dimension slicing?

A matrix multiply produces each cell of its result as a sum of products taken along an index the two inputs share — the contraction dimension. Contraction-dimension slicing computes that sum over a selected subset of those positions instead of all of them, so the result is approximate but far cheaper. In RMM the subset is chosen at inference from the input passing through, and the paper states the model's weights are left unchanged, so no retraining is involved.

Why does it matter that RMM is training-free?

Because it can in principle be applied to a checkpoint you did not train yourself, which is the situation most deployments are actually in. Structured pruning usually needs a finetuning run to recover accuracy, and quantization-aware methods need a calibration or training pass. A method that only changes how a matmul is executed leaves the model artifact alone, so it can be tuned with its retention ratio and turned back off. The summary does not say whether RMM's slice selection needs a setup pass of its own.

How does it relate to quantization and sparse attention?

All three reduce inference cost, but they cut different things. Quantization keeps every term in the sum and makes each number cheaper to store and multiply. Sparse attention and KV eviction keep full-precision math but skip selected query-key interactions, or drop cached entries entirely. RMM keeps the attention pattern and the number format and shortens each matmul's sum itself. They are largely independent, and the paper's ablation that attention-side computations are more reducible than MLP components is a hint about how to divide an error budget across them.

RMM cuts Transformer matmuls without touching the weights — Contraction-dimension slicing

TL;DR

What is it: The news anchor is an arXiv paper proposing Reduced Matrix Multiplication (RMM); the concept it makes concrete is contraction-dimension slicing — at inference time, picking out the informative slices along the shared index that a matrix multiply sums over, and skipping the rest, with the model's weights left exactly as they were.
Why it’s needed: Most of a Transformer's inference cost is matrix multiplication, and the paper reports that the fraction of that sum you keep is a continuous dial rather than a fixed setting — a retention ratio you can turn per deployment, and per input, without retraining anything.
vs previous: Where quantization makes each number cheaper, pruning deletes weights permanently, and sparse attention skips selected query-key interactions, RMM leaves the weights, the attention pattern and the layer count untouched and instead shortens the sum itself — which is why it needs no finetuning and why its knob moves smoothly instead of in steps.

Think of it as

a long grocery receipt you total by reading only the lines that move it

one line on the receipt = one slice along the contraction dimension, the shared index a matrix multiply sums over
the total at the bottom = one cell of the result matrix, which is nothing but that sum
reading only the lines that move the total = RMM selecting informative slices at inference and skipping the rest
a different shop on a different day putting different lines in play = input-adaptive selection, decided per input rather than baked into the weights
how far down the list you bother to read = the retention ratio, a continuous dial between accuracy and compute
the snack lines rounding away where the rent line cannot = the paper's finding that attention-side computations are more reducible than MLP components

Jargon

Contraction dimension: In a matrix multiply, the index that both inputs share and that disappears from the result. Multiply a matrix of shape M by K with one of shape K by N and you get M by N: K is the contraction dimension, and every cell of the answer is a sum of K products. It is the loop the naive matmul step walks from end to end.
RMM (Reduced Matrix Multiplication): The paper's method: choose a subset of the K slices and compute the sum over just those. The paper describes it as training-free and input-adaptive — the model is not retrained and its weights are not modified, and the choice is made from the activations flowing through at the time.
Retention ratio: The fraction of the contraction dimension that survives. The paper presents it as the control that produces a smoother accuracy-efficiency tradeoff, meaning you can ask for a small reduction and pay a small accuracy cost, rather than choosing between a few fixed configurations.
Training-free (post-hoc): Applied to an already-trained checkpoint rather than built into training. The paper describes RMM as training-free, with the weights left unchanged; the summary does not say whether a calibration pass is involved, so read it as "no retraining", not necessarily as "nothing to set up". The appeal is that it can in principle be turned on over a checkpoint you did not train — which also means it has to survive inputs nobody tuned it for.
MLP block: The feed-forward half of a Transformer block — two large matrix multiplies applied to each token independently, or three in the gated variants most current models use, holding most of a model's parameters. The paper's ablations contrast it with the attention half.
Attention-side computations: The matrix multiplies that belong to attention rather than to the MLP: forming queries, keys and values, scoring queries against keys, and mixing values by those scores. The paper reports these tolerate more reduction than the MLP does.
Arithmetic intensity: Floating-point operations performed per byte moved. It is the x-axis of the roofline plot, and it decides whether an operation is limited by the machine's math units or by its memory bandwidth — which is why where a saving lands matters as much as how large it is.
Ablation: An experiment that removes or varies one part of a method to see what that part was contributing. The claim that attention is more reducible than the MLP comes from ablations, not from the headline result.

The news. On August 13, 2026, researchers posted Reduced Matrix Multiplication (RMM), a training-free, input-adaptive method for reducing the matrix products a Transformer computes at inference. RMM selects informative slices along the contraction dimensions of those matrix products while leaving model weights unchanged, and exposes a retention ratio controlling how much computation remains — which the authors present as a smoother accuracy-efficiency tradeoff than discrete alternatives. The paper reports evaluation across language models from 1B to 70B parameters, with wall-clock benchmarks from custom kernels on an NVIDIA A100, and says the gains are largest at longer sequence lengths. Its mechanistic ablations find that attention-side computations are more reducible than MLP components. Read the paper →

Picture the receipt from a very long shop. Four thousand lines, one total at the bottom. If someone asks what the total is and you have thirty seconds, you do not add four thousand numbers — you run your eye down the column, add the lines that are actually large, and stop. The answer you get is not the true total, but it is close enough that the person asking cannot tell, and you paid for a few dozen additions instead of four thousand. Which lines were worth reading was not decided in advance; it was decided by looking at this receipt.

That is the shape of what RMM does, and the reason it has a place to stand is that a matrix multiply is built out of exactly this operation and nothing else. Multiply an M by K matrix by a K by N one and each of the M times N cells in the answer is a single sum of K products — one term for each position along the shared index. That shared index is the contraction dimension: the naive matmul step walks it from end to end for every output cell, reading two values per position — one from each input — so the length of that walk is what the cell costs.

C[2][3] reads row 2 of A and column 3 of B

■ row 2 of A■ col 3 of B■ C[2][3]

RMM's move is to stop walking all of it. Instead of summing every position along K, it selects the slices it judges informative for the input currently passing through, sums over those, and skips the others — and because the skipped work is a piece of the sum rather than a piece of the model, the weights are never modified and nothing is retrained. A retention ratio says how much of K survives. Turn it to 1 and you have the original matmul back; turn it down and you buy compute at a price in accuracy that the authors describe as moving smoothly rather than in steps.

Smoothness is worth dwelling on, because it is the practical difference from the neighbours. Quantization gives you 8-bit, 4-bit, sometimes 2-bit — a short menu of discrete points, and dropping a tier is a commitment. Structured pruning removes rows or columns for good, usually with a finetuning run behind it. A retention ratio is a continuous number, so a deployment can hold to a latency budget by moving it a little rather than by switching methods. The knob's granularity, not just its depth, is part of what is being claimed here.

Approach	What gets skipped	Are the weights touched?	What the knob looks like
Weight quantization	bits per stored number, not terms	yes — weights are rewritten in a smaller format	a short menu of formats (8 / 4 / 2-bit, setup-dependent, illustrative)
Structured pruning	whole rows or columns, permanently	yes — removed for good, usually with finetuning after	fixed once chosen; changing it means another prune
Sparse attention and KV eviction	query-key interactions, and with eviction, cached tokens	no	a budget of interactions or entries, resting on a prediction about which ones matter
Mixture-of-experts routing	whole expert blocks per token	no, but routing is trained in	discrete — a token gets a whole expert or none of it
RMM (contraction-dimension slicing)	positions along the shared index of each matmul	no — the paper states weights are left unchanged	a continuous retention ratio, applied per input

Here is where it earns its keep, in one matmul. Hold a single feed-forward projection fixed on a 70B-class model: hidden width 8,192, feed-forward width 28,672, weights in fp16. For one token that projection is a 1 by 8,192 activation against an 8,192 by 28,672 weight matrix, so its contraction dimension is K = 8,192 and it costs 2 x 8,192 x 28,672 = about 470 MFLOP, reading 8,192 x 28,672 x 2 bytes = about 470 MB of weights. Now price the same projection at a retention ratio of 0.5: the sum runs over 4,096 positions instead of 8,192, so it costs about 235 MFLOP and touches about 235 MB — because a skipped slice of the activation is also a skipped row of the weight matrix, which never has to be read. The saving lands on the arithmetic and the memory traffic at the same time, which is the property that makes it show up in wall-clock rather than only in a FLOP count. (This configuration and the 0.5 ratio are illustrative, built from the curriculum's own matmul arithmetic to show what a retention ratio prices; the paper reports its own wall-clock results from custom A100 kernels and does not publish these figures.)

1024×1024

FLOPs

2 × 1024³ = 2.1 billion

each output: 1024 multiply-adds

Bytes

3 × 1024² × 4B = 12.6 MB

load A + load B + store C (FP32)

≈ 170 FLOPs/byteCompute-bound

That last point is also the honest limit. Skipping arbitrary positions along K is only cheap if the kernel can actually decline to load them, and a general-purpose matmul kernel is built to stream contiguous memory rather than to hop over it — which is why the paper's wall-clock evidence comes from custom kernels on an A100 and not from a stock library call. The paper also reports the advantage growing at longer sequence lengths, and it is worth being careful about why: in the weight projections the contraction dimension is the model's hidden width, which does not move with sequence length at all. The one product whose contraction dimension really does grow with the sequence is the value mix — combining every earlier token's value vector, weighted by its attention score, which is a sum with one term per earlier token. Scoring queries against keys is not that: it contracts over the head width, and what grows with the sequence there is the size of the score matrix it produces, not the length of each individual sum. The summary reports the trend and not its cause, so treat that connection as a reading rather than a finding. A method that reads as pure arithmetic on paper is, in practice, a claim about kernels.

The finding most likely to outlive the paper is the one from its ablations, and it is a statement about where approximation is tolerable rather than how much of it you can afford. In the authors' words, "attention-side computations are substantially more reducible than MLP components." On the receipt, the snack lines round away and the rent line does not. If that asymmetry holds beyond this method, it is a budgeting rule for inference-time approximation generally: spend the error budget on the attention path and protect the MLP, rather than reducing every matmul by the same ratio because they look alike from the outside. Two limits are worth carrying alongside it — the evidence available here is the 24-page paper's own benchmarks, with no independent replication cited, and "informative slices" is doing real work that the summary does not unpack, so how the selection behaves on inputs unlike the evaluation set is exactly the question a deployment would need answered.

Goes deeper in: GPU & CUDA → Tiling & Matrix Multiply → Naive Matrix Multiply

Related explainers

Compute Where It Counts — per-token compute controller — the same instinct applied one level up: vary how much compute a token gets, rather than how much of a matmul's sum survives.
ThriftAttention — importance-aware FP16/FP4 attention — also spends precision where it matters, but by choosing a number format per region instead of shortening the sum.
I/O-optimal approximate attention — approximation judged by bytes moved rather than by operations skipped, which is the other half of why kernels decide these results.

Continue in trackTiling & Matrix Multiply: what the contraction dimension costs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based