What is NVGEMM in PyTorch 2.14?

NVGEMM is a new source of matrix-multiply kernels inside `torch.compile`. Instead of calling a prebuilt library entry point, CuTeDSL generates a CUTLASS kernel specialized to your shape and dtype, and Inductor benchmarks it against the Triton and ATen candidates for the same operation.

What is a fused epilogue, and why does it help?

The epilogue is the element-wise work right after a matmul — a bias add, an activation, a rescale. Run as a separate kernel it forces the output tile out to HBM and back for almost no arithmetic. Fusing it runs that work while the result is still in registers, so the tile is written once instead of three times.

Does NVGEMM replace Triton or ATen?

No. It is added as a third candidate, not a replacement. Inductor autotunes across all three and keeps whichever is fastest for the shape in front of it, so ATen still wins where the vendor kernel is well matched and Triton still wins where its fusion reaches further.

How does the compiler decide which kernel to use?

By measurement rather than heuristic. Inductor builds the candidate kernels, times each on the actual input shape, and caches the winner. That is why the same model can select different backends for different layers, and why the choice can change when you alter a batch size or sequence length.

GPU· 2026-09-06

PyTorch 2.14 ships NVGEMM kernels — NVGEMM epilogue fusion — What does it mean?

TL;DR

What is it: PyTorch 2.14, released on September 2, 2026, adds NVGEMM — a third source of matrix-multiply kernels inside torch.compile, generated from CUTLASS templates by CuTeDSL — whose headline capability is epilogue fusion: folding the bias add, activation and rescale that follow a matmul into the matmul kernel itself.
Why it’s needed: A matmul's epilogue is arithmetically trivial but expensive to move — run it as its own kernel and the entire output tile takes a full round trip out to HBM and back, spending the resource that is actually scarce on a modern GPU while performing about one add per element.
vs previous: Until now Inductor chose between Triton-generated kernels and ATen's prebuilt cuBLAS calls; NVGEMM adds a third candidate that is generated per shape, and because all three are benchmarked on your actual dimensions it addresses the failure mode where a backend fixed at library-build time is only fastest for the shapes its author happened to measure.

Jargon

GEMM: General Matrix Multiply — the dense A @ B + C operation that dominates the arithmetic in both training and inference. Almost every layer of a transformer is one or more GEMMs.
Epilogue: The cheap element-wise work that runs on a GEMM's output immediately after the multiply: adding a bias, applying an activation such as GELU or SiLU, or rescaling a quantized result.
ATen: PyTorch's built-in operator library. For a matmul it dispatches to a prebuilt vendor kernel, normally NVIDIA's cuBLAS — well tuned, but chosen when PyTorch was built rather than for your shapes.
Triton: A Python-embedded language for writing GPU kernels at block granularity, so you describe tiles rather than individual threads. torch.compile generates Triton kernels for you — see How Triton Works.
CUTLASS and CuTeDSL: CUTLASS is NVIDIA's open template library of GEMM building blocks; CuTeDSL is the Python DSL that emits them. Together they let a compiler generate a matmul kernel specialized to one shape and dtype instead of calling a fixed library entry point.
Inductor: The backend of torch.compile that turns a captured graph into actual GPU kernels. It decides what to fuse, which kernel source to use, and — through autotuning — which candidate wins.
Autotuning: Building several candidate kernels for the same operation, timing each one on the real input shape, and keeping the fastest. It replaces a static heuristic with a measurement, at the cost of compile time.
NVFP4: A 4-bit floating-point format on NVIDIA Blackwell tensor cores, used with per-block scale factors so a whole tensor is not forced onto one exponent. Full explainer

The news. On September 2, 2026, PyTorch released version 2.14, landing 2,995 commits from 487 contributors. Inductor gains CuTeDSL-generated NVGEMM kernels, which are autotuned alongside Triton and ATen and support fused epilogues and scaled or NVFP4 GEMMs. The release also ships a complete nccl2 collective backend with nonblocking communicators and eager split, in-place process-group repair — the release notes call it the point at which "Fault tolerance becomes a first-class c10d concept" — CUDA graph capture for torch.while_loop, Rubin sm_107 support, and Python 3.15 in eager mode. Read the release →

Picture the bakery. The loaf is one matrix multiply — say a 4096 x 4096 projection inside an attention block — and the shop has three ovens that can bake it. The stock oven is ATen: it came with the shop, NVIDIA tuned it, and it is genuinely excellent at the loaf sizes they had in mind. The second oven is Triton, which you assemble yourself from a short recipe describing tiles and staging; torch.compile writes that recipe on your behalf. NVGEMM is the third oven, and it is built to the shape of this particular loaf — CuTeDSL emits a CUTLASS kernel specialized to your dimensions, your dtype, and the work that comes after the multiply.

The interesting part is not the third oven. It is that the shop no longer guesses which oven is best — it bakes the loaf in all three and keeps the fastest. This is autotuning, and it matters because the winner flips on details you cannot reliably reason about from the source: the exact matrix dimensions (M, N and K), whether those dimensions are multiples of the tensor-core tile, how much of the weight matrix stays resident in cache. A backend fixed once at library-build time is only fastest for the shapes its author happened to measure. Inductor compiles the candidates, times them on your real shapes, and caches the answer — the same measure-rather-than-guess discipline you meet in torch.compile.

Now the glaze. A matmul almost never ends at the multiply: there is a bias to add, a GELU or SiLU to apply, sometimes a scale factor to fold back in for a quantized model. That work is arithmetically trivial and ruinously expensive to move. In the bakery it means carrying every finished loaf across the room to a glazing bench and carrying it back. On a GPU it means the output tile leaves the streaming multiprocessor that produced it, is written out to HBM, and is read straight back in by a second kernel that performs almost no arithmetic. A fused epilogue glazes the loaf inside the oven — the bias and the activation run while the result is still sitting in registers, before it is ever written out. That is operator fusion, and what NVGEMM changes is that CUTLASS epilogues become available on a generated matmul rather than only on a hand-written kernel.

Unfused (3 kernels)

HBM (read)

↓

matmul

↓

HBM (write+read)

↓

bias add

↓

HBM (write+read)

↓

ReLU

↓

HBM (write)

6 HBM accesses

Fused (1 kernel)

HBM (read)

↓

matmul
+ bias
+ ReLU

↓

HBM (write)

2 HBM accesses

3× fewer HBM accesses — same computation

Put numbers on the round trip. Take one 4096 x 4096 output tile in bf16 (16-bit brain float, 2 bytes per number — see Precision Formats) — illustrative, since the release publishes no benchmark figures of its own. The tile holds 16,777,216 elements at 2 bytes each, so it is 33.6 MB. Unfused, that tile is written out by the matmul, read back in by the bias-and-activation kernel, and written out once more: three touches, 100.7 MB of HBM traffic. Fused, it is written exactly once: 33.6 MB. On a GPU with 3 TB/s of memory bandwidth (illustrative) that is 33.6 µs versus 11.2 µs — and the arithmetic bought by the two extra touches is one add and one activation per element. The multiply was never the problem; the round trip was. That ratio is why fusion shows up everywhere once you start looking for it, and why the roofline model puts an element-wise epilogue firmly on the memory-bound side of the ridge.

Kernel source	Where it comes from	Epilogue fusion	The catch
ATen	Prebuilt vendor library (cuBLAS / cuBLASLt), fixed when PyTorch was built	Fixed menu — only the epilogues the vendor exposes	No specialization to your shapes
Triton	Generated by Inductor from a Python block-level recipe	Yes, for pointwise chains Inductor can prove are safe to fuse	Block-level abstraction, so the lowest-level tensor-core scheduling is the compiler's call
NVGEMM (new in 2.14)	CuTeDSL generates a CUTLASS kernel specialized to the shape	Yes — CUTLASS epilogues, including scaled and NVFP4 GEMMs	One more candidate to generate and benchmark, so compile time grows

Goes deeper in: GPU & CUDA → Operator Fusion & FlashAttention → Operator Fusion

Related explainers

INT8 finally beats FP8 on consumer GPUs — Fused INT8 GEMM kernel — the same epilogue trick pushed into quantization, with dequantization and bias folded into the GEMM instead of run as a separate pass.
Atrex-Bench tests LLM-written kernels on production traces — what goes wrong when candidate kernels are timed on the wrong shapes, which is exactly the failure autotuning is trying to avoid.
vLLM 0.25.1 stops a fused kernel from corrupting NVFP4 models — the other face of fusion: a fused kernel that was silently wrong for one dtype combination.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based