NVIDIA replaces GenAI-Perf with multiprocess AIPerf — Multiprocess load generation — What does it mean?

The news. On September 18, 2026, NVIDIA published its introduction to AIPerf, the designated successor to GenAI-Perf and a ground-up rewrite that benchmarks LLM inference from a single-GPU server up to multi-node deployments. Worker processes generate load, separate record-processor services handle results, and ZMQ coordinates them, so the client does not go GIL-bound under real concurrency. It covers 15+ endpoint types, replays production traces from Mooncake, Baseten and WEKA AgentX, shapes arrivals as constant, Poisson or gamma, and reports TTFT, ITL, request latency and output-token throughput at p25 through p99 with optional DCGM or pynvml GPU telemetry. One section heading carries the whole idea: "The client shouldn't be the bottleneck." Read the announcement →

Picture the hydrant test. You want to know how much water the hydrant can move, so you screw a garden hose onto it, open the valve all the way, and watch the meter on the hose settle at twenty litres a minute. You write down twenty. The hydrant can do two thousand. Nothing about the measurement was dishonest — the meter read the hose exactly right — but the hose, not the hydrant, is what set the number.

A benchmark client is that hose, and the narrow part is often not the network. For every request the client has to build the payload, hold the connection open, and then sit inside the response stream timestamping each token event as it arrives, because that is the only way to produce TTFT and inter-token latency at all. Nearly all of that is Python-level work, and CPython's GIL lets one process run Python bytecode on one core at a time. Threads do not rescue you; they queue behind the same lock. So a single-process load generator has a ceiling on the request rate it can actually offer. NVIDIA's announcement puts the tool it is replacing squarely there, describing GenAI-Perf as a single-process architecture that goes GIL-bound once concurrency or request rate gets real.

Under-reported throughput is the obvious damage. The subtler one is that a saturated client can also inflate the latency it reports. The timestamp a client writes for "first token" is written when its own event loop gets to the socket, not when the token landed on the wire, so a client running at its ceiling folds its own queueing into the number you are about to act on. That distortion is hardest to spot in the tail, because delayed reads look exactly like real tail events — a p99 that is partly your own scheduler is a p99 you cannot separate from the server's.

AIPerf's answer is to split the hose into a manifold. Worker processes do nothing but generate load, each one a separate interpreter with its own GIL; separate record-processor services do the aggregation into percentiles; ZMQ carries records between them. The division of labour matters more than the process count: the part that is sending requests never stops to compute a p99, so the client's ceiling scales with processes instead of staying pinned to one.

The dials above are the server's half of the picture: push the arrival rate toward capacity and latency goes vertical. A capped client never gets the rate high enough to find that knee, so the curve you plot can look flat and safe right up to the day production finds the knee for you. Put numbers on the cap and it falls out of arithmetic. Hold two things fixed: each request streams 500 output tokens and raises one event per token, and the client spends about 20 microseconds of CPU per streamed token event — parse the chunk, stamp the clock, hand it to the recorder. That is 500 × 20 microseconds = 10 ms of client CPU per request. One Python process gets one core for that work, so its ceiling is 1000 ms ÷ 10 ms = 100 requests per second, before it has spent a cycle building payloads or opening sockets. Give it four worker processes on four free cores and, if they scale cleanly, the ceiling moves to roughly 400 req/s. Now say the server's real capacity is 300 req/s: the single-process run reports 100 and calls the server saturated; the multiprocess run reports 300. Same server, same prompts, three times the answer — the first number was the hose and the second is the hydrant. (Figures illustrative; the announcement publishes no per-token client cost.)

Process count is only half of what a load generator decides on your behalf. The other half is the shape of the load. A benchmark that sends requests at an exactly even rate is producing a traffic pattern production systems rarely produce, and evenly spaced arrivals remove one source of queueing — arrival clustering — though overload and service-time variation still produce queues on their own. AIPerf exposes three shapes (announcement):

Same mean rate, different queueing, different tail — which is how goodput, the rate of requests that actually meet their latency target, collapses while throughput looks untouched. The practical test covers both failures. Add worker processes until the reported number stops moving; everything below that plateau was your client. The plateau itself is a ceiling, not proof it is the server's — cross-check it against GPU utilization before you believe it. Then change the arrival shape at the same mean rate and watch whether the number moves again; if it does, you were measuring an assumption about traffic rather than a property of the server. The same discipline applies one layer up, where an agent's metrics pipeline is also a client that samples and aggregates in-process, and can also become the thing that limits what it can see.

Goes deeper in: LLM Serving → Serving Metrics → Capacity Planning & Saturation

Related explainers

• Trajectory-replay benchmarking — the same move one layer up: replay recorded traffic instead of synthesising it.
• Controller-worker separation — why splitting the part that decides from the part that works keeps reappearing.
• Load-dependent latency model — what a measured latency actually depends on once load stops being fixed.