The news. On August 25, 2026, Microsoft published a paper describing Maia 200, an inference accelerator built as a software-defined, locally accessed dataflow machine. The stated figures are 10,145 TFLOP/s at FP4, 5,072 TFLOP/s at FP8, 7 TB/s of HBM bandwidth, and a 750W TDP — thermal design power, the power budget the cooling is built for. The described mechanism is a change of who is in charge of data: software explicitly programs dataflow engines that coordinate specialized memories and movement engines, rather than scheduling a thread-centric execution model. Multilevel DMA and a hierarchical network-on-chip are described as keeping data movement predictable, and narrow-precision datapaths are aimed at the compute and bandwidth profile of reasoning-model inference. Read the paper →

Picture a workshop where every bench worker fetches their own parts. When a worker needs a bracket, they put down their tools, walk to the parts store, find the bin, and walk back. A floor manager stands in the middle deciding, moment to moment, who gets to go next when two workers want the same aisle. It works, and it is flexible — a worker who suddenly needs a different bracket simply walks and gets it. But nobody can tell you in advance what the floor will look like in ten minutes, and a worker at the bench with empty hands is a worker producing nothing. Now change one thing. A planner writes a timetable before the shift begins: this bracket arrives at bench four at minute six, that casting at bench seven at minute nine. The parts move on a conveyor to a schedule written before the shift starts, and nobody walks anywhere.

The first workshop is a GPU. You write a kernel, it runs as thousands of threads grouped into blocks, and the hardware assigns those blocks to the units that will run them. On a GPU, the decision about what runs next is made by hardware, at runtime — the scheduler picks which group of threads occupies an execution unit this cycle, and it does so partly on the basis of which ones are stalled waiting for memory. That responsiveness is the design's great strength: it hides the cost of a slow memory read by running someone else meanwhile. It is also why the movement of data is a consequence of running threads rather than something the program states outright.

Maia 200 moves that decision into software: the control program sets up the movement instructions and their handshakes ahead of the run, and the dataflow program then executes asynchronously. The paper's description has three parts that fit together. Dataflow engines are the units software programs directly, and what it programs them to do is coordinate specialized memories and movement engines — memory that exists for a particular job, and hardware whose only job is carrying bytes between those memories. Multilevel DMA is how the carrying happens at every level rather than only between chip and main memory: dock to aisle bin to bench tray, not one long walk. And a hierarchical network-on-chip is the belt network those routes run on, tiered rather than flat. The word the paper uses for the result is predictable, and predictable is the point: a schedule you can write down beforehand is one whose costs you can count before you run it, instead of discovering them as stalls. The plan is not total, and the paper does not claim it is: where the work depends on the data — a mixture-of-experts router deciding at runtime which expert an activation goes to — a small control step evaluates the result and kicks off transfers that were configured in advance. The schedule is written ahead of time; what runtime decides is which of the prepared moves to start.

Why go to that trouble? Hold the chip's own published numbers still and the answer is arithmetic. Peak FP8 throughput is 5,072 TFLOP/s and HBM bandwidth is 7 TB/s, so to keep the math units saturated from memory the chip needs about 725 FLOPs of work for every byte delivered — that ratio is the balance point between the two stated figures, and at FP4 (10,145 TFLOP/s against the same 7 TB/s) it roughly doubles, to about 1,450. Now price the work. Generating one token at a time uses each weight for about two operations, a multiply and an add, so a batch-of-one decode step supplies roughly 2 FLOPs per byte of weights read — around 360× short of the 725 it would take to keep FP8 arithmetic busy. (The 2-FLOPs-per-weight figure is the standard convention for a matrix-vector product, not a number from the paper; real serving batches many requests and lifts this considerably.) The chip's scarce resource is data movement, not arithmetic — which is precisely the resource this architecture hands to software.

Who decides	Thread-centric GPU	Maia 200 dataflow
What the program describes	threads and the work each one does	dataflow engines and the movement between memories
When movement is decided	at runtime, as threads issue requests	set up ahead of the run; data-dependent cases start preconfigured transfers
How stalls are hidden	by switching to another ready thread group	by overlapping the scheduled movement with computation, asynchronously
Data path between memories	caches and the memory system, largely implicit	multilevel DMA over a hierarchical network-on-chip
Stated peak arithmetic	varies by part (setup-dependent, illustrative — not from this paper)	10,145 TFLOP/s FP4 · 5,072 TFLOP/s FP8, at 750W

The tradeoff is the one every static schedule makes. A timetable is only as good as the planner's model of the shift, and work whose shape is not known until it runs — a branch that depends on a value, a request whose length nobody knew in advance — is exactly what a runtime scheduler is good at and a compiled schedule is not. Inference is an unusually good fit for the timetable because its shape repeats: the same layers, in the same order, for every token. The paper does not rest on peak figures alone: on Qwen 2.5 7B, a straightforward PyTorch implementation calling standard kernels reaches 2,434 tokens per second, which the authors put at more than 70% of the estimated maximum — the fraction of the ceiling that survives a real model is the number a planned schedule is trying to raise, and that one is high. What the paper deliberately withholds is a head-to-head against a GPU on the same workload; the authors say they refrain from one on purpose. So the evidence supports the narrower claim — a compiled schedule keeps this machine close to its own ceiling — and leaves the wider comparison, against the machine you are running today, open.

Goes deeper in: GPU & CUDA → Execution Model → Mapping to Hardware: SMs

Related explainers

• Cerebras CS-5 — Wafer-scale memory locality — the other way to attack data movement: change where the data sits, rather than who schedules its trip
• NVIDIA pairs Rubin GPUs with Groq 3 LPX — Phase-split inference — prefill and decode have opposite bottlenecks, which is why one chip's schedule cannot suit both phases
• Vera Rubin NVL72 — The rack-scale NVLink domain — the same predictability argument applied between chips instead of inside one

Check what you knowMap your AI & GPU knowledge across every track — free, role-based→