TPUv7 Ironwood serving benchmark — Data-parallel attention vs expert parallelism — What does it mean?

The news. On September 7, 2026, SemiAnalysis published third-party serving results for Qwen3.5-397B at FP8, comparing Google's TPUv7 Ironwood against NVIDIA B200 and B300 on an 8k-input / 1k-output workload. At 20 tokens/s/user it reports 9,364 total tokens/s/chip on Ironwood versus 8,903 on B200 and 8,925 on B300 — roughly a 5% throughput edge. The wider gap is modelled cost: at 100 tokens/s/user, $0.181 per million total tokens on Ironwood against $0.222 (B200) and $0.276 (B300). The native TorchTPU stack behind the numbers is still in private beta. Read the benchmark →

Picture the wards first. Each chip in the pod is a ward, it admits its own patients, and every patient's chart stays at that bedside. Nobody photocopies a chart for the ward down the hall, because nobody down the hall is treating that patient. That is data-parallel attention: each chip owns a distinct slice of the in-flight requests, and each request's KV cache lives on exactly one chip.

The reason the hospital is arranged that way is a number in the model card. Qwen3.5 has two KV heads. Tensor parallelism splits an attention layer by head — give each rank its own heads, and each rank does part of the same work — but two heads do not divide across a wider group of chips (the report does not state its group size). The usual workaround is to replicate: every chip keeps a full copy of every request's keys and values. Replication spends the whole group's cache memory on copies of the same keys and values, and the KV cache is normally the thing that limits how many requests you can hold at once — so memory that could have bought concurrency buys nothing. Grouped-query attention shrank that cache on purpose; splitting it by head then hands the saving back.

Now the clinics. The same model carries 512 routed experts, and the report has them distributed across the pod under expert parallelism, one subset per chip. A token that needs a particular expert has to travel to whichever chip holds it. So inside one forward pass the attention layers want requests held still and the expert layers want tokens moved around, and the serving engine has to satisfy both at once. The walk between them is an all-to-all: every chip hands a different bundle of tokens to every other chip, the experts run, and the results come back the same way. The engine that places requests and keeps the block tables straight is doing it across two parallelism axes that disagree.

Where the microseconds actually go. The benchmark's quieter finding is about the walk, not the wards. SemiAnalysis reports that combining two metadata all-gathers saved roughly 80 microseconds per layer in its DeepSeek-V3 measurements. To get a feel for the size of that, hold the depth fixed at 60 layers and compose it out — illustrative throughout, because the source gives a per-layer saving measured on a different model, and states neither a layer count nor a decode-time breakdown for Qwen3.5. One forward pass would give back 60 × 80 µs = 4.8 ms, and the interactivity target the benchmark is run at is 20 tokens per second per user, a 50 ms budget per token. Those two quantities land within about a factor of ten of each other. That is a statement about scale rather than a measured share of anyone's decode step — but it is the right order of magnitude to notice, because none of it touches a single matmul. The other collective they tuned points the same way: a hierarchical SparseCore ReduceScatter — the collective that sums a value across chips and hands each chip back one slice of the result — lifted 8k1k throughput by 4.1% to 14.2% across concurrency 64 to 512, and that one is measured directly. When attention and the experts sit on different parallelism axes, the shuffle between them is not overhead around the real work — it is a large enough share of the step that rewriting one collective moves the benchmark.

One caution about reading any single row of this benchmark: the layout above is why the chip keeps up on this workload, not proof it keeps up on every workload. The report says so itself — "TPUv7 does not have native FP4 computation, thus on FP4, NVIDIA GPUs still maintain the lead." A parallelism layout is a property of the model and the memory you have; the precision the silicon supports is a separate axis, and it does not move because the collectives got faster.

Goes deeper in: Inside vLLM → Distributed Execution → Three Axes, One Point Each

Related explainers

• PrivateUse1, and the kernels PyTorch did not compile — how ordinary PyTorch reaches a non-NVIDIA accelerator at all, and why some kernels stay hand-written