Racer is a symbolic equivalence checker for GPU kernels from Northeastern University. It runs a kernel and a reference on symbolic inputs at the PTX level, turns each output cell into an exact expression, and checks whether the two agree for all inputs.

Why is differential testing not enough for GPU kernels?

It samples inputs and accepts small differences. The paper identifies four blind spots: values that sampling rarely produces, state that is not an argument such as uninitialized memory, outputs the test never inspects, and logic errors smaller than the tolerance.

Does Racer prove that a kernel is correct everywhere?

No. A result of Equivalent holds for the checked launch configuration and for exact real arithmetic, so it does not cover other shapes or rounding behavior. Racer can also return Unknown when neither the algebra system nor exact refutation can decide.

How does it relate to how GPUs execute threads?

A launch is thousands of threads grouped into CTAs that synchronize at barriers, so Racer must model barriers, shared memory and atomics. It steps threads in lockstep between barriers and refuses cases it cannot order.

GPU· 2026-10-22

Racer finds 82 GPU kernel bugs — symbolic equivalence checking — What does it mean?

TL;DR

What is it: Racer, a paper from Northeastern University, is a tool that proves an optimized GPU kernel computes exactly the same function as its reference for every input under the checked launch configuration, using exact real arithmetic and symbolic equivalence checking instead of running sample tensors.
Why it’s needed: GPU kernels are rewritten for every chip generation, number format and serving setup, and each rewrite must keep the function unchanged, but today that promise rests almost entirely on tests that compare sampled outputs within a tolerance.
vs previous: Differential testing runs both kernels on random tensors and accepts small differences, so it misses boundary values, uninitialized memory, stray writes and logic errors smaller than the tolerance; Racer turns every output cell into an exact expression and checks all inputs at once.

Jargon

PTX: NVIDIA's stable virtual instruction set. CUDA C++ and Triton kernels both compile down to it, so Racer can analyze kernels written in different languages at the same level.
CTA: Cooperative thread array, the formal name for a thread block: a group of threads that share fast memory and can wait for each other at a barrier. See Threads, Blocks, and Grids.
Differential testing: Run a new kernel and a trusted reference (usually PyTorch) on random input tensors and compare the outputs element by element within a tolerance.
Tolerance: The allowed gap between two outputs, set as an absolute and a relative amount. It exists because a correct kernel that adds numbers in a different order gives slightly different rounding.
Symbolic execution: Running a program on named variables instead of numbers, so each instruction builds an algebraic expression such as c0x0 - s0x1 rather than computing a value.
CAS: Computer algebra system, software such as SymPy that rewrites expressions exactly. Racer asks it whether the difference of two expressions simplifies to zero.
Shape Reduction: Racer's way to prove one representative output cell and reuse that proof for every cell with the same expression structure, instead of proving each cell alone.
Exact numeric refutation: When the CAS cannot decide, plug in one concrete rational assignment and evaluate both expressions exactly. A nonzero difference proves the kernels differ; a zero difference proves nothing.

The news. On October 20, 2026, Tuan Truong Vu and Cheng Tan of Northeastern University posted Racer, a symbolic equivalence checker for GPU kernels. Across 2,408 production kernels from 41 collection targets it found 82 real bugs in 23 libraries, 76 of them previously unknown, and 41 were confirmed when the paper was written. Read the paper →

Picture two spreadsheets that are supposed to compute the same totals: one is the trusted reference, the other is the fast version someone rewrote. The usual check is to type a few sample numbers into both and see whether the totals look close. Racer does something else: it leaves the input cells blank, lets each spreadsheet show the formula it would apply, and subtracts one formula from the other. If the difference simplifies to zero, the two sheets agree for every number you could ever type in.

In Racer, that idea becomes symbolic equivalence checking over a chosen launch configuration. Racer runs the kernel and its reference on symbolic inputs, so every output cell ends up holding an exact algebraic expression in the input variables. Because a single free variable stands for all values at once, one check covers the entire input space, and because the expressions are exact there is no tolerance to tune. If you want the hardware background first, the execution model lesson covers threads and blocks, and warps covers how 32 threads move together.

Why sampled tests keep missing bugs

The authors studied the 82 bugs Racer found and sorted each by the main reason a test could overlook it. A test compares numbers, but correctness is a statement about functions, and that gap shows up in four ways.

Blind spot	Example from the paper	Why a sampled test misses it
Sampled input	attorch's hardshrink keeps an input that equals the threshold λ exactly, where PyTorch drops it	Inputs on either side of the boundary are correct, and random tensors rarely land exactly on it
Implicit input	A Triton-distributed reduction atomically adds into an output cell that was allocated but never zeroed	Memory happens to hold zeros during testing, so the sum looks right
Unchecked output	fla's squared-ReLU backward writes all 4,096 results of a block even when the output has only 64 elements	The test compares only the declared tensor, and its 64 values match
Below tolerance	FlagGems' InstanceNorm adds ε where it should subtract it when recovering the variance	The wrong value differs by less than the tolerance that correct kernels need

The paper counts seven findings in the sampled-input group, 11 in the implicit-input group, six out-of-bounds writes in the unchecked-output group and six below-tolerance bugs. The InstanceNorm case matters because a systematic error like that can compound over millions of training steps, as the authors point out. Read the bug study.

How Racer reaches a verdict

Racer takes two kernels, the tensors they read and write, and a launch configuration, then returns one of three verdicts: Equivalent, Inequivalent with a concrete input that separates the two, or Unknown. It works in four steps.

Read the checking instance, a small Python class that declares the shared input tensors and each program's launches.
Lower both programs to PTX, so CUDA C++ and Triton kernels are compared at the same level.
Execute every thread of every CTA symbolically, which leaves one expression in each output cell.
Subtract the paired expressions cell by cell and ask the CAS whether each difference is zero.

Stopping at "Unknown" is a design choice. Racer fails closed: if it cannot prove or refute a cell, or cannot order the threads, it refuses to answer instead of guessing.

Three techniques that make it practical

A naive version of this idea is far too slow. The paper reports that a naive implementation of symbolic equivalence did not finish within a 600 second budget even for a simple Softmax kernel on a 4 × 64 tensor. Racer adds three techniques, each answering one problem.

Shape Reduction groups output cells whose expression pairs match under a renaming of input variables, proves one representative, and reuses the proof for the rest. On one linear-attention backward kernel it cut end-to-end time from 204.9 s to 17.4 s at the captured shape, and at four times the size whole-grid checking exceeded 600 s while Shape Reduction finished in 70.7 s.
A lockstep CTA scheduler advances every thread to the next barrier before continuing, records reads and writes in epochs between barriers, and models asynchronous copies as available only after their wait. It refuses what it cannot order, so no verdict depends on an interleaving the engine happened to pick.
Idiom templates and exact numeric refutation recognize compiled routines such as log, replace them with the math they stand for, and evaluate stubborn cells at a concrete point, with a Z3 solver as a last resort for boundary cases like x at or above λ versus x strictly above λ.

A worked example: the rotary-embedding bug

The paper's running example is vLLM issue 941. Fix two variables: T = 2 tokens and head dimension D = 4. The reference rotates the pairs (x0, x1) and (x2, x3), the GPT-J layout. The shipped kernel pairs (x0, x2) and (x1, x3), the NeoX layout. Both run the same arithmetic and differ only in which addresses they load. For token t, output cell 0 holds c0·x0 − s0·x1 in the reference and c0·x0 − s0·x2 in the shipped kernel. Their difference is s0·(x1 − x2), which is not identically zero. Substitute s0 = 1, x1 = 1 and x2 = 0 and it equals 1. One assignment gives a difference of 1, so Racer returns Inequivalent with that witness, and no sampling was needed.

Now the below-tolerance case, with illustrative numbers. If the variance is 1.0 and ε is 0.00001, adding ε where the code should subtract it overestimates the variance by 2ε, which is 0.00002, a relative error of 0.002 percent. A test tolerance chosen to forgive reordered floating-point sums can easily be larger than that, so the test passes; the symbolic difference is exactly 2ε, which is not zero, so Racer flags it.

What the result does and does not show

Against the 82 bugs it found, the authors assess an oracle differential tester with unlimited compute that could in theory detect 47, NVIDIA's compute-sanitizer detects 18, and Volta, the closest existing equivalence checker, detects 9. This is a per-bug assessment on the bugs Racer itself found, not a measured testing campaign, so treat it as a sign of which blind spots matter and not as a ranking of tools.

Three limits are worth keeping in mind. Each check is tied to one launch configuration, so a pass covers every input but not every tensor shape. Arithmetic is interpreted over the real numbers, which is why rounding differences vanish from the comparison; this suggests a pass shows two kernels compute the same function, not that they round identically. And the comparison needs a reference, which in the paper's pipeline was an independent implementation, an LLM-drafted naive reference, or another configuration of the same kernel. Where the checker returns Unknown, an Unknown alone does not establish a bug, and six of the 82 came from manual inspection of such cases. For how number formats change rounding in the first place, see precision formats.

Goes deeper in: GPU & CUDA → Execution Model → Threads, Blocks, and Grids

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based