What is PyTorch's PrivateUse1 backend?

It is a dispatch slot PyTorch reserves for hardware it does not ship support for. A vendor registers its accelerator against that key, and operations on that device route through PyTorch's normal dispatcher, so `tensor.to(device)` works like any other device move — without the vendor's code living inside PyTorch. The slot is the entry point, not the implementation: the backend still has to supply the operations.

What are Pallas kernels, and why keep them if there is a compiler?

Pallas is a Python dialect for writing TPU kernels by hand at tile level, roughly what Triton is for NVIDIA GPUs. The report says TorchTPU retains TPU-specific Pallas kernels alongside the compiled path, without saying which. The general reason a stack does this is that a compiler is strong on average and not always the best available code for the few operations that dominate a hot loop.

How is this different from PyTorch/XLA?

PyTorch/XLA has historically reached TPUs through lazy-tensor tracing: operations are recorded rather than executed, and the graph runs when something forces it. That makes tensors behave differently from eager PyTorch. TorchTPU instead registers through PrivateUse1 and reports ordinary eager tensors, using `torch.compile` plus StableHLO for the graph path.

Does this make a TPU a drop-in replacement for a GPU?

Not on its own. The software path removes the rewrite, but hardware differences remain — the same report notes TPUv7 has no native FP4 computation (a 4-bit float format the newest NVIDIA parts execute directly), where NVIDIA keeps the lead — and the serving win in the benchmark also depends on the parallelism layout the model allows, not only on the framework integration.

GPU· 2026-09-10

TPUv7 Ironwood serving benchmark — PrivateUse1 backend with retained Pallas kernels — What does it mean?

TL;DR

What is it: The TorchTPU stack behind SemiAnalysis's TPUv7 Ironwood benchmark reaches the chip through PrivateUse1, the device slot PyTorch keeps free for a new accelerator, then lowers compiled graphs through StableHLO and XLA — while retaining hand-written Pallas kernels outside the compiled path.
Why it’s needed: Without a native path into the framework, a new accelerator forces backend-specific rewriting of code that already works, so the integration route decides whether a chip inherits the existing PyTorch ecosystem or spends years catching up to it.
vs previous: The long-standing route to a TPU was PyTorch/XLA's lazy-tensor tracing, which records operations instead of running them, so a TPU tensor did not behave like an ordinary eager one — reading a value mid-function forces the accumulated graph to execute; TorchTPU instead registers the device through PrivateUse1, PyTorch's dispatch slot for out-of-tree hardware, and reports ordinary eager tensors on the chip.

Jargon

PrivateUse1: A dispatch slot PyTorch reserves for out-of-tree backends. A vendor registers its hardware against that key and routes operations through PyTorch's ordinary dispatcher without patching PyTorch itself — though the backend still has to implement the operations it claims to support.
FX graph: The graph torch.compile captures from your Python before optimizing it — the handoff point between what you wrote and what the backend compiles. See torch.compile.
StableHLO: A portable, versioned operation set that sits between a framework and a hardware compiler. It is the common language the graph is translated into, so the backend never has to understand PyTorch's internals.
XLA: Google's compiler, which turns that operation set into machine code for a TPU (or a GPU), fusing operations along the way.
Pallas: A Python dialect for writing TPU kernels by hand at tile level — the TPU's counterpart to Triton.
Kernel: One program that runs on the accelerator. A framework operation may become one kernel or many; fewer and larger usually runs faster, which is why fusion is a compiler's main job.

The news. On September 7, 2026, SemiAnalysis published third-party Qwen3.5-397B FP8 serving results for TPUv7 Ironwood against NVIDIA B200 and B300, run on Google's native TorchTPU stack. The headline is economic — at 100 tokens/s/user it models $0.181 per million total tokens on Ironwood against $0.222 on B200 — but the enabling detail is the software path: ordinary PyTorch tensors, on a TPU. TorchTPU is still in private beta, with open-sourcing expected around mid-October. Read the benchmark →

Start at the empty station. PyTorch keeps a slot on the shelf labelled for a guest — a device key nobody owns, that any vendor may claim and fill. A backend registered in that slot is reached through the ordinary dispatcher, so an ordinary .to(device) call can land on it — and it does not have to persuade PyTorch's maintainers to merge anything. The slot does not do the cooking, though. It is an entry point, and the backend still has to implement what it claims to support. That slot is PrivateUse1, and native in TorchTPU's description is the result of having done that work: the report describes ordinary PyTorch tensors on a TPU.

Then the recipe book. Once torch.compile has captured your Python into a graph, that graph still describes PyTorch operations, and a TPU compiler has no reason to know what those are. StableHLO is the translation step: a portable, versioned operation set that the framework writes and the hardware compiler reads, so neither side has to track the other's internals. XLA is the line cook — it takes the book and produces machine code for the chip, fusing operations as it goes.

And then the dishes nobody hands to the book. The report says only that TorchTPU retains TPU-specific Pallas kernels alongside the compiled path; it does not say which ones, or why. The general reason a stack keeps hand-written kernels is familiar enough, though: a compiler is strong on average and is not always the best available code for the few operations that dominate a hot loop. Pallas is the TPU's Triton: a Python dialect for writing tile-level kernels by hand, for people who do not want to write the machine's native low-level code. The shape worth noticing is the hybrid one — a compiled path plus retained hand-written kernels, rather than all of one or all of the other — which is the shape vLLM arrived at on NVIDIA hardware too, where an attention backend is a pluggable object carrying its own kernels.

GPU Abstraction Stack

PyTorch EagerL4

Write Python ops — runs immediately, one kernel per op

torch.compileL3

Annotate with @torch.compile — compiler fuses & optimizes for you

TritonL2

Write tile-level kernels in Python — full control, no C++ needed

CUDA C++L1

Write thread-level kernels — maximum control, maximum complexity

↑ easier↓ more control

What the plug is worth. Put the two modelled costs side by side at the benchmark's 100 tokens/s/user point: $0.181 per million total tokens on Ironwood against $0.222 on B200 — a gap of $0.041 per million, or about 18%. Now fix the volume at one trillion total tokens a month (illustrative — the report models cost per token, not anyone's traffic). That is 1,000,000 million-token units, so the gap comes to 1,000,000 × $0.041 = $41,000 a month. None of that is reachable by a team whose framework cannot target the chip, and that is the whole point of the PrivateUse1 route: the cost advantage is a hardware-and-cost-model result, but it only becomes a purchasing decision if ordinary PyTorch code runs there unmodified. The stack is the precondition for the number, not a footnote to it.

Route to new hardware	How the tensor behaves	What the vendor maintains	What it costs
Fork the framework	native — but only inside the fork	a whole parallel PyTorch	permanent merge debt against upstream
Tracing / lazy-tensor layer	deferred; not ordinary eager semantics	a tracing layer plus a compiler	reading a value forces the graph to run
PrivateUse1 registration	eager, once the backend implements the operations	a backend, kept out of tree	must supply that operation coverage itself
Hand-written kernels only	no framework integration at all	one kernel per operation	does not scale past a few models

Goes deeper in: Inside vLLM → Attention Backends & the Platform Layer → Adding a Backend

Related explainers

Why attention went data-parallel while the experts did not — the parallelism layout the same benchmark runs on, and what two KV heads rule out

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based