What is trace-weighted kernel benchmarking?

It is scoring generated GPU kernels on tasks pulled from real serving traces, with each task weighted by how much GPU time it actually consumes. Atrex-Bench samples operators and shapes from full-cluster inference traces — 30 operators and 440 shapes — and applies importance weights tied to observed GPU time and serving phase. The effect is that an operator the cluster spends most of its time on counts for more than one it rarely touches, so the final score reflects the fleet's real workload rather than the shape of a hand-built task list.

Why score kernels against a roofline instead of a baseline?

Because a baseline is only as good as whatever it was compared against, while a roofline is anchored to the machine. The roofline is the ceiling the hardware imposes, set by whichever of compute or memory bandwidth saturates first, so scoring against it turns "faster than before" into a fraction of what was possible. It also removes a hiding place: a PyTorch fallback is correct, so a check that only asks whether the code is right cannot tell it apart from a real kernel, but against a roofline the same submission simply reports how much of the machine went unused. Atrex-Bench reports that the best vanilla model reaches only about 10% of the hardware roofline.

How does this relate to the CUDA expertise gap?

It measures it. The promise of a model that writes kernels is that it closes the gap between people who can write CUDA and people who need fast kernels, and the honest question is how far it has actually closed. Atrex-Bench answers with a fraction of roofline on production-shaped work rather than a pass rate on puzzles, and its companion Atrex-Kernel-Agent attacks the gap with iterative measure-revise search, optimization dropout, and a knowledge base of 298 reference-kernel files and 244 optimization-knowledge documents — moving, in the paper's terms, from fallbacks toward real kernels.

Atrex-Bench tests LLM-written kernels on production traces — Trace-weighted kernel benchmarking

TL;DR

What is it: The news anchor is Atrex-Bench, a benchmark that tests LLM-written GPU kernels on operators and shapes sampled from production inference traces; the concept it makes concrete is trace-weighted kernel benchmarking — scoring each task by the GPU time it actually burns in serving rather than counting every task equally.
Why it’s needed: A kernel that is correct and a kernel that matters are different things, and the gap is measurable: Atrex-Bench reports that the best vanilla model reaches only about 10% of the hardware roofline, a number an unweighted pass-rate would never surface.
vs previous: The established shape is the synthetic programming puzzle — hand-built kernel tasks, each counted once — which can be passed by a PyTorch fallback that is correct but never becomes a real kernel; Atrex-Bench weights each problem by observed GPU time and serving phase and scores against a roofline ceiling, so a fallback scores like the fallback it is.

Jargon

Kernel: A single program that runs on the GPU — the actual code that does one step of the math, like a matrix multiply. Writing a fast one means managing where the data sits and how the threads read it, which is why it is a specialist skill and why people want a model to do it.
Operator and shape: The operator is what the math is (an attention step, a matmul, a normalization). The shape is how big the tensors are when it runs. The same operator at a different shape can want a different kernel, which is why Atrex-Bench samples both — 30 operators and 440 shapes, per the paper.
Roofline: The ceiling on how fast an operation can possibly run on a given chip, set by whichever runs out first: compute or memory bandwidth. It turns “is this kernel fast?” into “how close is it to the fastest this hardware allows?” Read a roofline plot step by step →
Importance weight: A multiplier on each benchmark task set by how much GPU time that task really consumes in the traces. A task the cluster spends most of its night on counts for more than one it touches rarely. This is the mechanism that makes the score reflect the fleet rather than the task list.
Serving phase: Whether work happens during prefill (reading the prompt, usually compute-bound) or decode (emitting tokens one at a time, usually memory-bound). The same operator sits in a different place on the roofline in each. See the split →
PyTorch fallback: When the generated code just calls the stock PyTorch operation instead of a hand-written kernel. It is correct, and it runs, so a check that only asks whether the code is right can pass it — but no kernel was authored. The paper’s framing is a gap between syntactic correctness, fallback behavior, and real kernel performance.
Measure-revise search: The loop the companion Atrex-Kernel-Agent runs: write a kernel, measure it, revise from what the measurement said, repeat. The paper names the loop; it is the measurement, rather than a compiler accepting the code, that drives the next revision.
Optimization dropout: One of the pieces the paper names inside that loop, alongside the knowledge base, as part of how the agent moves from fallbacks toward real kernels. The source names it without spelling out its mechanics — the name suggests holding some optimizations back so the search does not commit to one path too early, but that reading is an inference, not a claim from the paper.

The news. On July 16, 2026, the Atrex-Bench paper landed: a benchmark of production-shaped GPU kernel tasks rather than synthetic programming puzzles. It samples operators and shapes from full-cluster inference traces, weights each problem by observed GPU time and serving phase, and includes roofline ceilings so a kernel is scored against what the hardware allows rather than against a baseline. It reports 30 operators and 440 shapes, and that the best vanilla model reaches only about 10% of the hardware roofline. A companion Atrex-Kernel-Agent uses iterative measure-revise search, optimization dropout, and a layered knowledge base of 298 reference-kernel files and 244 optimization-knowledge documents. Read the paper →

Picture two ways to grade a kitchen. The first is a cooking exam: a fixed set of dishes, each scored once, marked on whether the plate came out right. The second is to walk in the morning after a real service, pull the actual tickets off the rail, and grade the cook on those — and, crucially, on how many minutes of the night each one really ate. Both are tests. Only one of them can tell you whether dinner went out. The exam grades the cook against the task list; the tickets grade the cook against the night the restaurant actually had.

That gap is the whole idea, and swapping the kitchen for a GPU changes almost nothing. A synthetic kernel benchmark is the exam: a curated set of problems, each counted once, marked on whether the generated kernel is correct and beats a baseline. Atrex-Bench is the ticket rail. It samples operators and shapes from full-cluster inference traces, so the task list is a record of what the fleet actually ran, and then it does the part that matters: it weights each problem by the GPU time that problem really consumed, and by the serving phase it ran in. A benchmark that counts every task once is quietly asserting that every task matters equally, which is the one thing a production trace can prove false.

The second move is what the kitchen calls the burner and the paper calls the roofline. Ask "is this kernel fast?" and you need something to be fast relative to — and a baseline is a moving target, only as good as whatever it was compared against. The roofline is a far more stable one: it is the ceiling the chip imposes, set by whichever of compute or memory bandwidth runs out first. It is still a modeled ceiling — you have to say which hardware limits you are counting — but it is anchored to the machine rather than to a rival implementation. Grading against it converts a vague "faster than before" into a fraction of what was possible. That is what makes the paper's headline number legible rather than merely bad: the best vanilla model reaches only about 10% of the hardware roofline — not ten percent slower than a baseline, but a tenth of what the silicon was sitting there willing to do.

And it is the roofline that exposes the microwave. The failure the paper puts its finger on is the PyTorch fallback: generated code that simply calls the stock operation instead of writing a kernel. It is correct. It runs. It plates on time. And that is exactly the problem — a check that asks only whether the code is right has no way to tell a kernel from a passthrough, because the passthrough is right. Scored against the roofline, the same submission has nowhere to hide: the ceiling does not care who wrote the code, only how much of the machine went unused. The paper's real subject is the distance between three things a single pass-rate blurs into one — code that is syntactically correct, code that falls back, and code that actually moves the hardware.

Work the weighting to see how much it can move a score. Fix two operators, one kernel each, and hold the agent's actual work constant (illustrative — the split below is invented; only the 10%-of-roofline figure is the paper's). Say the agent writes a genuinely good kernel for a rare operator, hitting 80% of roofline, and falls back on the operator that dominates the night, landing at the paper's 10%. Grade it as an exam — every task once — and the score is the plain average: (80 + 10) / 2 = 45% of roofline. Now grade it on the tickets. Suppose the traces say the rare operator is 5% of GPU time and the dominant one is 95%. The weighted score is 0.05 × 80 + 0.95 × 10 = 13.5%. Same two kernels, same agent, same hardware — 45% becomes 13.5% purely by asking which of the two the cluster actually spends its night on. The weighting did not measure the kernels differently. It measured a different question.

What the benchmark decides	Synthetic kernel puzzles	Atrex-Bench
Where the tasks come from	hand-built problem set	operators and shapes sampled from full-cluster inference traces
How much each task counts	once — every task equal	importance-weighted by observed GPU time and serving phase
What a kernel is scored against	a baseline implementation	a roofline ceiling
What a PyTorch fallback earns	it is correct, so a check that only asks whether the code is right cannot tell it from a real kernel	scored against the roofline ceiling, it reports as the fraction of the machine left idle
Reported scale	varies by benchmark	30 operators, 440 shapes

The companion Atrex-Kernel-Agent is the paper's answer to its own bad news, and it is built out of the same insight. Its loop is iterative measure-revise search — write a kernel, measure it, revise from what the measurement said — which is the benchmark's philosophy turned into a feedback signal: what drives the next attempt is a measurement, not a compiler's approval. Around that loop sit optimization dropout and a layered knowledge base of 298 reference-kernel files and 244 optimization-knowledge documents. The paper names optimization dropout as part of this machinery without spelling out its mechanics, so what it does in detail is not something this article can tell you. The paper describes the result as moving from fallbacks toward real kernels — which is precisely the axis the trace weighting and the roofline ceiling were built to make visible in the first place. You cannot climb a gap you cannot see, and the benchmark's first job was to make the gap a number.

Goes deeper in: GPU & CUDA → Roofline Model → Reading the Roofline Plot

Related explainers

INT8 finally beats FP8 on consumer GPUs — Fused INT8 GEMM kernel — what a hand-written kernel actually has to get right to approach the ceiling this benchmark measures against
NVIDIA Blackwell leads AgentPerf — Trajectory-replay benchmarking — the same instinct one layer up: score the system on replayed production work, not a synthetic script
Agent leaderboards mislead under distribution shift — Predictive validity — the general form of the failure here: a benchmark that scores well but predicts the wrong thing

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based