KernelBench study of a local 7B model — Correct first, fast second — What does it mean?

The news. On November 9, 2026, researchers at National Taiwan University posted Correct First, Fast Second (arXiv 2611.12079). They held one locally hosted model, Qwen2.5-Coder-7B-Instruct, fixed and changed only the interaction around it: zero-shot prompting, profiling-guided prompting, up to three retries on the error log, and a multi-agent loop in which the same model plays analyzer, researcher, kernel generator and feedback triager around a compile-and-run evaluator. They ran the first 10 tasks of each KernelBench level (30 in all) on an NVIDIA V100 in FP32. Read the paper →

Picture the junior cook in their first week. The head chef's signature dish — a large matrix multiply or convolution — is already served by cuBLAS and cuDNN, NVIDIA's tuned GPU libraries, and on large dishes the junior does not beat it: in every regime, correct kernels averaged 0.97–0.99× the speed of eager PyTorch, and the "correct" matrix-multiply attempts only tied by calling torch.matmul again. A large square matrix multiply is compute-bound — limited by arithmetic, not by memory traffic — and sits far right on the roofline plot, where the only lever is raw arithmetic and the library already uses it fully.

The junior's real wins are at the finishing station: fusing the bias, activation and clamp steps after a convolution into one pass, which is exactly where the roofline says custom code can win. Each separate elementwise op reads and writes the whole tensor from GPU memory, so it is memory-bound, and fusing the chain into one kernel removes memory round-trips and kernel launches. The agent's best speedup, 1.23×, came on a matrix multiply with a very small inner dimension, where cuBLAS is held back by launch overhead and low utilization rather than by arithmetic.

Now add managers who keep telling the junior to attempt the hardest dish. The pilot agent's prompts asked every role to write custom tiled CUDA on every task, and the 7B model could not compile what it was asked for: it called PyTorch's load_inline (which compiles CUDA source from Python) with a keyword argument that does not exist and mismatched module names. Each retry, guided by a critic that asked for more optimization, stayed in the same over-ambitious mode, and the selection logic could replace an earlier working kernel with a later broken one. That pilot scored 3% correct and 33% compile — below plain zero-shot's 30% correct — and a single-turn version of it beat the full loop. A retry loop only helps when each turn can actually move toward a correct answer.

The repair had four parts, each aimed at one observed failure:

• Competence-matched prompts: reach a correct, compiling kernel first — reuse torch ops for the heavy compute, fuse only the memory-bound epilogue — and escalate to hand-written tiling only after that. The prompts also pin the exact load_inline signature the model kept inventing.
• Best-of-n sampling: 4 candidates per turn at temperature 0.7, ranked by the real evaluator.
• Monotone turn selection: turns are ranked by correct, then compiled, then speedup, so a broken retry can never displace a working kernel.
• A 240-second hard timeout per candidate, in a child process, because one generated kernel deadlocked the GPU and stalled a run for 14 hours.

Switching off one part at a time shows which part carries the load: switch off best-of-n and the repaired agent falls back to 3% correct, so the prompts alone do not rescue a weak generator — the taster checking several plates does. Removing the feedback analyzer actually raised correctness by 7 points, because its rewritten diagnoses over-specified the next attempt, while the raw compiler error left room for a small fix.

No regime wins everywhere, and the split follows task difficulty. On level 1 (one operator), simpler is better: the agent's analyzer talked the model into custom kernels where a one-line torch op passes, and it dropped to 5 of 10. On level 2 (a chain of 3–5 operators), the agent is the only regime that solves any task, because naming the exact operator chain and its data flow before writing code, plus a second chance after a semantic miss, is what fusion needs. On level 3 (whole networks), the profiling-guided prompt wins by pointing the model at the dominant layer. The authors estimate that sending levels 1 and 3 to the guided prompt and level 2 to the agent would reach 53% correct on this set — a routing decision, not a smarter model.

Here is the compute trade in numbers. Hold the model, the 30 tasks and the three-turn agent fixed, and vary only n, the number of candidates per turn. At n = 1 the agent is correct on 3% of tasks (about 1 of 30), and only 23% even compile. At n = 2 correctness jumps to 27% (about 8 of 30): +24 points for one extra candidate per turn. At n = 4 it reaches 30% (9 of 30), and at n = 8, 33% (10 of 30) — the last doubling buys +3 points, one more task, for twice the evaluation cost. Compare the profiling-guided prompt, which uses the same compute as zero-shot and lifts correctness from 30% to 43% (about 9 to 13 tasks) only by adding the top operators by CUDA time and one roofline rule: do not reimplement matmul or conv, fuse the epilogue. For a small model on a checkable task, the cheapest gains come first from telling it what not to attempt, then from extra candidates the oracle can check — not from more reasoning about one candidate.

Two limits bound the result. It is one 7B model, one seed and 30 tasks, and the authors say gaps of one or two problems are within noise, so they put weight on the large effects — the 3% to 30% repair, the 0-versus-3 split on level 2, the best-of-n curve — rather than on small gaps. It also ran in FP32 on a V100, which has no FP32 tensor cores; on newer GPUs the library ceiling may sit elsewhere, and the competence-matching claim is made for small models only. The infrastructure lesson applies more widely: a Python signal cannot interrupt a running CUDA kernel, so anyone evaluating generated GPU code unattended needs each candidate in its own process with a hard wall-clock kill.

Goes deeper in: GPU & CUDA → Roofline Model → Where ML Operations Fall

Related explainers

• Atrex-Bench tests LLM-written kernels on production traces — why a correct PyTorch fallback should not score like a real kernel
• AMDKernelVault trains an 8B model to write AMD GPU kernels — the same correctness-before-speed ordering, built into an RL reward instead of a prompt
• PyTorch 2.14 ships NVGEMM kernels — the compiler doing epilogue fusion for you
• LARK formally verifies LLM-written GPU kernels — proving a kernel correct instead of testing it on random inputs