CUDA Multi-Process Service is a daemon that routes the CUDA work of several processes through a single GPU context, so their kernels run on the streaming multiprocessors concurrently instead of each process taking exclusive turns. It is a binary-compatible alternative implementation of the CUDA API, so applications do not have to change.

Why does CUDA MPS matter for inference serving?

Small models leave most of the GPU idle on any one request — AWS reports 15-20 percent utilization for a 0.6B speech model — and time-slicing does not hand that headroom to anyone else. Running several instances concurrently raises the sustainable in-flight count, which is what raises throughput within a fixed latency budget.

How is CUDA MPS different from MIG?

Both let one GPU serve several workloads, but MIG cuts the hardware into fixed physical partitions with their own memory controllers, while MPS keeps one shared context and divides the SM budget between clients. MIG gives stronger isolation; MPS gives a finer, reconfigurable split and is the better fit when the tenants trust each other.

Does CUDA MPS make individual requests faster?

Not directly. MPS raises how many requests a GPU can hold at once rather than speeding up any single request's kernels, so the throughput gain is a concurrency effect — Little's Law rather than a faster forward pass. The AWS post does not publish a like-for-like single-request comparison with and without MPS, so treat any per-request latency change as unmeasured here rather than as zero.

GPU· 2026-09-04

Cut ASR GPU infrastructure 75% with CUDA MPS — Concurrent execution vs time-slicing — What does it mean?

Jargon

SM (streaming multiprocessor): The GPU's actual unit of execution. A GPU is a bank of SMs — 142 on the L40S in this post — and a kernel occupies as many as its thread blocks fill. How blocks land on SMs
Time-slicing: CUDA's default way of sharing a GPU between processes: each one gets exclusive access to the whole device for a slice of time, and the others wait. Fine when one process fills the GPU, wasteful when it does not.
CUDA MPS: Multi-Process Service. A daemon that funnels the CUDA work of many processes through a single GPU context, so their kernels run concurrently rather than in rotation. It is a binary-compatible alternative implementation of the CUDA API, so the application code does not change.
MIG (Multi-Instance GPU): The third way to share a GPU: hard physical partitions, each with its own memory controllers, giving fault and performance isolation as well as memory protection. Stronger isolation than MPS, but the split is fixed rather than a share of the SMs.
CUDA_MPS_ACTIVE_THREAD_PERCENTAGE: The environment variable that sets how much of the GPU one MPS client may use. Heidi runs transcription as four instances at 25% each, and its speaker-labelling model as eight at 12%.
Triton Inference Server: NVIDIA's serving layer. It batches and routes requests across model instances; here each Triton model instance is mapped onto one MPS partition, so the scheduler and the GPU split line up.
p99 latency: The latency that 99% of requests finish at or under, while the slowest 1% run longer. A mean can look healthy while that tail blows the latency target, which is why capacity here is set by p99. Percentiles and tail latency
CUDA graph: A recorded sequence of GPU kernel launches that can be replayed as one unit, removing per-launch overhead. It has to be captured first, and that capture is the step MPS makes delicate. How capture and replay work

The news. On August 27, 2026, AWS published a walkthrough of how Heidi Health — which the post says handles over 2.4 million clinical consultations a week across 190 countries — serves speech recognition on EC2. Its Parakeet TDT 0.6B V2 model uses only 15-20 percent of a GPU per request, but CUDA's default time-slicing hands each process the whole device in turn, leaving about 80 percent of the hardware idle. Running Triton Inference Server on top of CUDA MPS, the team reports 92.1 requests per second (RPS) per GPU while still meeting its latency target, and recommends 4 GPUs where 16 run today — a 75 percent reduction. Read the post →

Picture the kitchen. Ten burners along one wall, and every order that comes in needs two of them. The waste is not the cook's fault — it is the house rule that only one cook may stand at the stove at a time. The cook lights two burners, eight stay cold, and three more cooks wait at the door holding orders the kitchen has the capacity to make right now.

That house rule is CUDA's default. A GPU is not one processor but a bank of streaming multiprocessors — 142 of them on the L40S the post benchmarks — and a kernel occupies only as many as its work fills — which is what a thread block actually gets scheduled onto. A speech-recognition request on a 0.6B model is small work: the post measures it at 15-20 percent of those 142 SMs, a fifth of the stove. But time-slicing gives each process the whole device for its slice, so the burners nobody lit are not handed to the next order — they simply stay cold until the slice ends. The GPU is not busy; it is merely occupied.

Your code creates:

Grid

256 threads

→GPU
assigns

GPU hardware runs:

SM0

B0B3

SM1

B1B4

SM2

B2B5

You create blocks — the GPU decides which SM runs each one

CUDA MPS changes the rule, not the kitchen. An MPS daemon funnels the CUDA work of every client process through one shared GPU context, so their kernels are resident on the SMs together instead of rotating through them. The post weighs the three sharing mechanisms NVIDIA ships for this, and they differ in what gets divided: time-slicing divides the clock, MPS divides the SM budget, and MIG divides the silicon itself.

Which one is right follows from a single question — is one process able to saturate the device on its own? If it is, time-slicing costs nothing and MPS adds a daemon for no gain. If it is not, every slice is mostly idle hardware, and that is the case MPS was built for.

Approach	What gets divided	Isolation	Wins when
Default time-slicing	The clock — processes rotate, each holding the whole device for its slice	Each process has the device to itself while its slice runs	One process can already saturate the GPU
CUDA MPS	The SM budget — all clients share one GPU context, each capped to a percentage of active threads	Separate address spaces, so memory is protected; the shared context is not fault-isolated	Each process uses only a fraction of the SMs
MIG	The silicon — hard physical partitions with dedicated memory controllers	Hardware-enforced, including fault and performance isolation	Tenants must not be able to affect each other

Hold the latency budget fixed — a mean under 650 ms and a p99 under 1,000 ms — and the benchmark reads as a ladder. With 8 requests in flight the GPU returned 48.7 RPS at a 166 ms mean; at 16 in flight, 78.2 RPS at 206 ms; at 32 in flight, 92.1 RPS at a 352 ms mean and a 769 ms p99 — the last rung that still clears the budget. Little's Law is the sanity check on those three columns: throughput is roughly the number of requests in flight divided by their latency, so 32 ÷ 0.3525 s ≈ 90.8 RPS, within about one and a half percent of the measured 92.1. (The law is exact for the average number of requests actually in the system; a closed-loop benchmark's configured concurrency is a close stand-in, not the identical quantity.) Push to 64 in flight and throughput inches to 99.7 RPS while the mean jumps to 659 ms — past the budget, so the extra concurrency buys capacity you are not allowed to count. MPS earns its keep by moving the rung you are allowed to stand on, not by making any single request faster.

L=λ×W

L5in-flight= computed

λ50req/s

W0.10seconds

Derived:

λ50 req/s

Little's Law: L = λ × W is exact for any stable queue. Service time fixed at 50 ms. As λ approaches capacity (~100 req/s), W diverges — that's the knee in the saturation chart on the right.

Concurrency is never free, and the post is unusually honest about the bill. MPS keeps each client in its own address space, so one process still cannot read another's memory — but they share a GPU context, and that is where the trouble lives. Under MPS a CUDA graph recapture triggered by an unseen tensor shape can be corrupted by a sibling instance, crashing that process with an illegal-address error. So the team warms up the shapes it expects and falls back to eager execution for the rest, staggers model loading behind a file lock, and runs a wedge sentinel health check that catches an instance that has become unrecoverable. What MPS gives up is not memory safety but fault isolation: the neighbours cannot read your data, they can still take you down with them — and it is CUDA graph capture, which assumes it owns the device while it records, that finds the seam first.

Goes deeper in: GPU & CUDA → Execution Model → Mapping to Hardware: SMs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based