ZYFA is a set of FP8 prefill-attention kernels for the AMD Instinct MI355X GPU, published by Zyphra in November 2026. It combines a one-multiply-add FP8 probability, a ping-pong schedule between two waves, and per-block power-of-two scaling, and reports 1.16–1.76× the throughput of AMD's AITER kernels on captured model inputs.

Why is FP8 attention harder to speed up than an FP8 matrix multiply?

FP8 matrix instructions run at twice the BF16 peak rate, but attention also contains a softmax that runs on the vector units, which FP8 does not accelerate. As the matrix work shrinks, the softmax takes a larger share of each block's time, so the kernel must overlap the two almost perfectly.

How does ZYFA keep FP8 attention accurate?

It scales each block of probabilities up by a power of two before rounding so that fewer small probabilities round to zero, and it divides by the sum of the rounded probabilities so the weights still sum to one. At head dimension 128 the paper reports numerical error 62–78% lower than AITER's.

How does this relate to FlashAttention?

ZYFA keeps FlashAttention's blockwise loop with a running softmax and changes how the work inside each block is scheduled and rounded. The paper notes that FlashAttention-3 calls its two-wave alternating arrangement ping-pong scheduling.

GPU· 2026-11-07

ZYFA FP8 attention on AMD MI355X — power-of-two probability scaling and ping-pong softmax — What does it mean?

Jargon

FP8 (e4m3): An 8-bit floating-point format with a 4-bit exponent and a 3-bit mantissa. It stores far fewer distinct values than BF16 but lets matrix hardware run at twice the BF16 peak rate.
MFMA: The matrix instruction on AMD GPUs, the counterpart of an NVIDIA tensor-core operation. It multiplies small tiles and runs on the matrix pipe.
Vector pipe: The part of a GPU core that runs elementwise instructions such as add, multiply and exponential. Softmax runs here, and FP8 does not make it faster.
Wave: A group of threads that issue instructions together on AMD hardware, like a warp on NVIDIA. Two waves share each SIMD (one execution unit) in this kernel.
Prefill: The phase where the model reads the whole prompt at once. At long context, attention often dominates its cost.
AITER: AMD's library of inference kernels. It is the baseline ZYFA is compared against.
Binade: A range of values that share one exponent, so each binade spans a factor of two. Counting binades below the maximum is a way to measure how small a probability is.
Underflow: A value too small for the format rounds to zero. In FP8, small attention probabilities underflow easily, and the keys they belong to silently drop out of the output.

The news. On November 5, 2026, researchers at Zyphra published ZYFA, FP8 attention kernels for the AMD Instinct MI355X. On attention inputs captured from real models they reach 1.16–1.76× the throughput of AITER, and 1.99× for DeepSeek-style multi-head latent attention on 32K-token Gaussian benchmarks. Read the paper →

Picture a kitchen with one stove and two cooks. The stove is the matrix pipe, the part of the GPU that multiplies tiles. The prep cook chops, which here means turning attention scores into softmax probabilities on the vector pipe. Switching from BF16 to FP8 doubles the stove's peak speed, but nobody gave the prep cook a faster knife. In FlashAttention the loop for each block of keys is matrix multiply, softmax, matrix multiply, so the stove stays cold whenever the chopping falls behind.

In FP8 the softmax needs about as many vector cycles as the matrix products need matrix cycles, so the two must overlap almost perfectly to keep the matrix unit busy. The paper reports that AITER reaches 39% of peak FP8 matrix throughput at long context, against 49% of peak in BF16. The MI355X adds a second difficulty: unlike NVIDIA Hopper and Blackwell, it does not track completion of matrix instructions in hardware, so the instruction order has to be set by hand in assembly.

Work per wave, per key block	FP8 cycles	BF16 cycles	Source
Two matrix products (matrix pipe)	1024	2048	ZYFA Table III
Softmax with exact P: FMA, exponential, convert (vector pipe)	896	768	ZYFA Table III
Softmax with lean P: FMA, integer convert (vector pipe)	384	not given	ZYFA Table III

The first fix is to make the prep cook's job much shorter. A standard FP8 kernel computes the probability with an exponential and then rounds it to FP8. ZYFA instead exploits how the FP8 format is built: the 8-bit pattern of an e4m3 number, read as an integer, is roughly a straight-line function of the base-2 logarithm of its value. That is the same observation behind Schraudolph's fast exponential. So the kernel computes the integer for the probability directly from the score with one multiply-add and one integer conversion, and the result is already the FP8 bit pattern. This is the cook eyeballing the cut instead of weighing each piece.

The price is accuracy. This "lean P" stays within 7.5% of the true probability after rounding, while rounding the exact value is within 5.9%, and it underestimates probabilities that are far below the row maximum. The paper then measures the effect on the model output rather than assuming it is harmless. The saving is large: the vector cycles per block fall from 896 to 384.

The second fix is the ping-pong schedule, a name FlashAttention-3 uses for this alternating arrangement. The two waves on each SIMD take turns: in one phase wave A issues the matrix instructions while wave B computes FP8 probabilities, and in the next phase they swap, with a barrier ending each phase. AITER uses this arrangement too, but its softmax phase runs longer than its matrix phase, so the matrix pipe sits idle for 31.7% of SIMD cycles while one wave waits at the barrier.

With lean P the softmax wave finishes before the matrix wave, so the matrix wave sets the pace of each phase. In the middle of the key loop the matrix pipe is busy for 86% of cycles with lean P, against 60% with exact P. Over the whole loop, including transitions between query tiles, the figures are 78% and 56%.

The third fix protects accuracy. A probability is at most 1 after subtracting the row maximum, so unscaled FP8 probabilities sit in the low part of the format's range, where values lose mantissa bits or round to zero. For every query and block of keys, ZYFA multiplies the block's probabilities by a power of two chosen from the gap between the block maximum and the row maximum, so the block's largest probability gets the code it would have had at the row maximum. The scaled matrix instruction on CDNA4 divides the factor back out as part of the product, so this costs no extra instruction.

The kernel also changes the denominator. If the numerator uses rounded probabilities and the denominator uses the unrounded FP32 sum, every key that underflowed to zero acts like a key with a zero value vector, and the output shrinks toward zero. ZYFA divides by the sum of the same rounded probabilities, so the attention weights still sum to one. It computes that sum with one extra matrix instruction per block, which adds 6% to the matrix cycles per block.

Here is the arithmetic on one wave and one key block, using the cycle counts in the table (measured by the authors for their setup). In BF16 the softmax needs 768 vector cycles against 2048 matrix cycles, which is 37.5% of the matrix time, so overlap does not have to be perfect. Moving to FP8 halves the matrix time to 1024 cycles, and exact P needs 896 vector cycles, which is 87.5% of it. Lean P cuts the vector work to 384 cycles, which is 37.5% again, the same headroom BF16 had. The cook can now finish chopping well before the stove is free.

One caution: the MI355X's sustained clock speed depends on the values being processed. Random Gaussian benchmark inputs therefore do not reliably predict speed on real activations, which is why the headline 1.16–1.76× comes from inputs captured from models and timed at sustained clocks.

Goes deeper in: GPU & CUDA → Tensor Cores & Mixed Precision → Precision Formats

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based