EFQ-Softmax (Exp-Free Quantization for Softmax) is a low-bit probability-generation method for attention. Instead of evaluating `exp()` in higher precision and quantizing the result, it maps shifted attention scores directly to block-scaled **E2M1** operands using a single affine rule per microscaling block.

Why does removing the exp() stage matter?

In low-bit attention the `QK^T` and `PV` multiplies already run on FP8 or FP4 matrix engines, so probability generation is a higher-precision stage sitting between them, and the `PV` multiply cannot start until it finishes. It runs on the vector unit, and the precision it resolves below the E2M1 grid is discarded at the handoff. The paper measures a **40.33% average reduction** in that stage's latency from 16K to 128K tokens on an A5 vector unit.

Does it change FlashAttention?

No. The FlashAttention-style row-maximum update, the historical rescaling of earlier partial sums, the high-precision accumulation and the final normalization are all left unchanged. Only the step that turns shifted scores into a `PV` operand is replaced, which is what makes it a drop-in rather than a new attention algorithm.

Does the lower precision cost accuracy?

Not in the reported evaluations. The Qwen3-8B seven-task mean rises from 0.6749 with MXFP4 to 0.6773, and the Qwen3-VL-8B-Instruct nine-task mean rises from 0.7826 to 0.8000. On WAN2.2-TI2V-5B the paper reports temporal consistency and visual quality comparable to the FP16 and MXFP4 baselines under VBench.

How does this relate to MXFP4 and other FP4 attention work?

MXFP4 is the baseline: a block of values shares one scale and each value is an E2M1 code. EFQ-Softmax keeps that representation but changes how the codes are produced. It is orthogonal to work like ThriftAttention, which decides *which* blocks deserve higher precision rather than changing the softmax path itself.

GPU· 2026-09-11

Replace exp-then-quantize softmax to cut vector latency 40.33% — Exp-free softmax quantization to E2M1 probability codes — What does it mean?

TL;DR

What is it: EFQ-Softmax, from a September 2026 paper, is a way to build the softmax stage of attention so it emits 4-bit E2M1 probability codes directly instead of computing exponentials in higher precision and quantizing the result afterwards.
Why it’s needed: Low-bit attention already runs the QK^T and PV multiplies on FP8 or FP4 matrix engines, which leaves probability generation as a higher-precision stage sitting between two low-bit multiplies — and the PV multiply cannot start until that stage hands over its operand.
vs previous: The previous approach, exp-then-quantize, evaluates exp() in higher precision, materializes a temporary probability block, then rounds it down for the low-bit PV multiply; EFQ-Softmax removes that round trip with one affine rule per microscaling block, fixing the producer/consumer precision mismatch between a high-precision softmax and a 4-bit consumer.

Jargon

E2M1: A 4-bit floating-point code: 1 sign bit, 2 exponent bits, 1 mantissa bit. It is what an FP4 matrix engine actually consumes.
MXFP4: Microscaling FP4: a small block of values shares one scale factor, and each value is stored as an E2M1 code. It is the baseline EFQ-Softmax is measured against.
Microscaling block: The small group of neighbouring values that shares a single scale factor. The scale is chosen from that block alone, so the block size sets how far one extreme value's influence reaches.
Shifted scores: Attention scores after the row maximum has been subtracted. Standard practice, because it keeps exp() from overflowing.
Exp-then-quantize: The conventional low-bit softmax path: exponentiate in higher precision, build a temporary probability block, then quantize that block for the low-bit PV multiply.
Vector unit: The part of an accelerator that runs elementwise math — exponentials, adds, comparisons — separate from the matrix engine that runs the large multiplies. Softmax lives here; QK^T and PV do not.
Online softmax: FlashAttention's trick of keeping a running row maximum and rescaling earlier partial sums as new blocks arrive, so the full score matrix is never written out. Full explainer
PV multiply: The second matmul in attention: the probability matrix times the value matrix. Its operand is exactly what the softmax stage has to hand over.

The news. On September 9, 2026, a paper titled EFQ-Softmax: Exp-Free Quantization for Softmax argued that low-bit attention has been optimized from both ends and squeezed in the middle. The QK^T and PV multiplies already run on FP8 or FP4 matrix engines, but the softmax between them still evaluates shifted-score exponentials in higher precision, forms a temporary probability block, and quantizes it before the low-bit PV multiply runs. That leaves a high-precision producer feeding a 4-bit consumer: a higher-precision stage wedged between two low-bit multiplies, which the second multiply has to wait for. Read the paper →

Stand at the deli counter for a minute. You point at a pile in the bin; the clerk lifts it onto a lab scale that reads 137.04 g, nods, and then tips the whole thing into a tub — and the tubs only come in a few fixed sizes. The weighing step produced precision that the tub threw away one second later.

That is the shape of the problem. The scale is the exp() call, run in higher precision. The tub is the E2M1 operand the FP4 matrix engine will accept. Everything the scale resolved below the nearest tub size is computed, stored, and then discarded — and while the clerk is weighing, the till is idle. On an accelerator the PV multiply cannot consume that operand until probability generation finishes — so the wait is spent on arithmetic whose answer will not survive the handoff.

The usual instinct is to make the scale faster. EFQ-Softmax does something else: it removes the scale. One pre-sized scoop fills the tub straight from the bin — a single affine rule that takes a shifted attention score and lands directly on a nonnegative E2M1 code, with no intermediate high-precision value to round.

Concretely, per microscaling block, EFQ-Softmax selects an exponent-only scale from that block's local maximum — the scoop sized once per bin, by the biggest pile in it — uses that scale to normalize the shifted scores, and maps them to nonnegative E2M1 codes with one affine rule. Exponent-only matters: a power-of-two factor is an exponent adjustment rather than a full multiply.

The same low-bit operand then updates both the PV numerator and the denominator, so both halves of the softmax are built from one representation of each probability. The paper states that consistency as a property of the method rather than as a measured defect of the baseline, so read it that way: it is what having one operand buys, not a bug it repairs. Reading two tickets off one scoop is simply a case that cannot disagree with itself.

The E2M1 name is the same scheme you already know from mixed-precision formats: E4M3 and E5M2 are the 8-bit members, E2M1 is two steps smaller — one sign bit, two exponent bits, one mantissa bit. Softmax output is nonnegative and bounded, which is part of why a direct mapping onto so small a grid is even on the table here.

What is most notable is the list of things EFQ-Softmax does not touch. The machinery that makes FlashAttention correct — the running row-maximum update, the historical rescaling of earlier partial sums, the high-precision accumulator, and the final normalization — is left exactly as it was. The change is confined to the one step that turns scores into an operand. That is what makes it a drop-in rather than a new attention algorithm: the surrounding loop, and the numerics of everything outside probability generation, are unchanged.

Stage	Conventional low-bit attention	EFQ-Softmax
`QK^T` multiply	FP8 / FP4 matrix engine	unchanged
Row maximum + historical rescaling	FlashAttention online update	unchanged
Probability generation	`exp()` in higher precision, materialize a block, then quantize	one affine rule, straight to E2M1
Block scale	chosen when the block is quantized	exponent-only, from the block's local maximum
`PV` numerator operand	the quantized probability block	the E2M1 codes
Denominator operand	not specified — not necessarily the same operand	the same E2M1 codes
Accumulation + final normalize	high precision	unchanged

What it measured

The paper reports end-to-end quality on three models and a separate kernel-level measurement. Together they answer two different questions — does quality survive, and does the kernel actually get faster.

Measurement	Baseline	EFQ-Softmax	Source
Qwen3-8B, seven-task mean	0.6749 (MXFP4)	0.6773	paper
Qwen3-VL-8B-Instruct, nine-task mean	0.7826	0.8000	paper
Fused probability-generation kernel, vector stage, A5 vector unit, 16K–128K tokens	—	40.33% average latency reduction	paper
WAN2.2-TI2V-5B under VBench	FP16 and MXFP4	comparable temporal consistency and visual quality	paper

Where that 40.33% actually lands

A percentage off one stage is not a percentage off the kernel, so walk it. Hold two things fixed: the QK^T and PV multiplies run on the matrix engine and do not move, and the accumulate-and-normalize tail does not move either. Only the vector stage changes. Suppose that at a 64K-token sequence the vector stage is 30% of the fused kernel's time (illustrative — the paper reports the stage's own reduction, not its share). Then 0.4033 × 30% removes 12.1% of the kernel's total time, and the other 70% is untouched.

Now run it backwards. To find those same 12.1 points anywhere else, that whole remaining 70% — matmuls and the accumulation tail together — would have to take 17.3% less time (12.1 / 70). The matmuls are already on the matrix engine in the format it was built for, and the paper leaves the accumulation tail unchanged on purpose. That is the shape of the lesson worth carrying off this paper: the cheapest speedup left in a well-tuned kernel is usually the stage you can delete, not the stage you can tune.

That is also how to read the quality numbers. Going from 0.6749 to 0.6773 is not the point, and neither is 0.7826 to 0.8000; the point is that they did not go down, and WAN2.2 stayed comparable to both baselines. On these three evaluations the latency did not cost accuracy — which is a statement about these models, not a guarantee that every workload comes out the same way.

Goes deeper in: GPU & CUDA → Operator Fusion & FlashAttention → FlashAttention: The Solution

Related explainers

ThriftAttention: importance-aware FP4 — the other way to spend precision in attention: reportedly keep the top ~5% of QK blocks in FP16 and drop the rest to FP4, instead of changing how probabilities are produced.
UFP4 and the E2M1 shrinkage bias — what goes wrong inside the E2M1 grid itself, and why its lopsided bins pull values toward zero.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based