AMD ROCm InferaSim — Sparse GPU anchors and transfer laws — What does it mean?
The news. In November 2026, AMD's paper described ROCm InferaSim, a tool that predicts LLM serving performance on AMD Instinct MI355X GPUs from a handful of short measurements, then searches those predictions for a good configuration. The authors report that 11 anchors cost 9.1 GPU-hours, while measuring the 226 fixed-length configurations those anchors cover took 204 GPU-hours. Read the paper →
Picture a restaurant kitchen. Every dinner service needs an oven to preheat for the same time whether one cook or eight are working, plus a pile of chopping that splits across however many cooks you hire. If you time one cook through a full service, you can predict a service with two cooks by keeping the preheat time and dividing only the chopping. That is the whole idea: measure one deployment, then compute the others instead of running them.
InferaSim sorts every serving choice into two bins. A choice that changes which GPU kernels run needs its own anchor: the model checkpoint, numeric precision, attention implementation, CUDA-graph mode, speculative-decoding method, engine and GPU type. A choice that only changes how much work each GPU does is predicted from an existing anchor: batch size, prompt and context length, TP, EP and PP widths, replica count, arrival pattern and a KV cache kept in host memory. The kernels that one engine step launches are called its kernel set.
An anchor is a short run on at most one node. It records the median decode step time at each batch size the engine captures as a graph, and the prefill time at five prompt lengths, fitted as T(n) = F + c1*n + c2*n^2 for n prompt tokens. The F term is the fixed per-step cost of kernel launches, synchronization, sampling and scheduling, and it stays largely unchanged when GPUs are added. A roofline-style model computed from hardware peak rates misses it, which is why it matters for capacity planning.
The transfer law keeps that fixed part and scales only the rest: new_step(b) = f + (old_step(b) - f) * rho. Here f is the batch-1 step from the anchor's own sweep, and rho defaults to the old width divided by the new width. On a DeepSeek-V4-Pro decode transfer from a TP4 anchor to TP8, halving the whole step gave a 44.1% median error, and the fixed-cost law gave 0.9% (batches 1 to 64, reported in the paper). Prefill behaves the same way: solving the TP4 and TP8 prefill steps for F gave 146.9 ms at 4k-token prompts and 145.9 ms at 8k, within 1%.
Here is the arithmetic with numbers held fixed (illustrative, not from the paper). A TP4 anchor shows a batch-32 decode step of 30 ms and a batch-1 step of 6 ms, so f = 6 ms and the divisible part is 24 ms. Moving to TP8 with the default rho = 4/8 = 0.5 gives 6 + 24 × 0.5 = 18 ms. Halving the whole step would say 15 ms, which is about 17% too fast, enough to make a deployment look like it meets its latency target when it does not.
The cost side uses the paper's own figures. Eleven anchors took 9.1 GPU-hours to cover 226 configurations that took 204 GPU-hours to measure, about 22 times less. For the larger searches in the study, the authors estimate that measuring every candidate would take about 25,000 GPU-hours.
| Approach | GPU cost to evaluate | Where it breaks |
|---|---|---|
| Measure every candidate | About 25,000 GPU-hours for the study's searches (paper) | Cost grows with the number of configurations; each trial holds its GPUs for a median 3.4 minutes plus engine start-up (paper) |
| Roofline arithmetic from peak rates | None | Ignores the fixed per-step cost; halving a 4-GPU DeepSeek-V4-Pro step to predict 8 GPUs gave 44.1% median error (paper) |
| Operator-level simulator | Profile every operator again whenever the model, engine or kernel library changes | Per the paper, Vidur models only dense layers and none of these simulators ships AMD GPU data (paper) |
| Anchors plus transfer laws | One measurement per kernel set; 11 anchors cost 9.1 GPU-hours (paper) | An anchor is matched by comparing fields, not by checking a kernel trace, so a wrong match is not detected |
A target larger than one node uses a three-step method the paper calls reduce, measure, restore. Shrink the target to a configuration that fits on one node, measure that, then add back what the reduction removed: pipeline stages, cross-node network hops, wider expert sharding, disaggregated pools and host-memory KV tiers. A discrete-event simulator then replays multi-turn agent traffic, with queueing, batching and prefix reuse, and a tuning agent searches the predictions. The output is a short list to confirm on real GPUs, not a final answer.
The reported accuracy is for AMD Instinct MI355X GPUs running vLLM, SGLang and AMD's ATOM engine. Over 172 configurations, median TPOT error was 8.1%. On a multi-turn coding workload, throughput error was within 8%. For 159 disaggregated deployments, none of them used as an anchor, throughput was within 5.5%. A five-candidate short list contained a deployment within 12% of the best feasible pairing of model and latency target. AMD also reports that each projection takes under half a second on one CPU core. These are the authors' numbers on one vendor's hardware, so treat them as a design pattern to copy, not a guarantee for another cluster.
Goes deeper in: LLM Serving → Serving Metrics & SLOs → Capacity Planning & Saturation