No Priors

Stefano Ermon on why diffusion models are built for the GPU

Stefano Ermon· Co-founder & CEO of Inception; Stanford professor at Inception
·~38 min·English·No Priors
LLMInferenceAI InfrastructureAI Company
TL;DR

Stefano Ermon, a pioneer of diffusion models and CEO of Inception, argues the next AI edge may come as much from fitting the model to the GPU as from making it smarter: diffusion generates tokens in parallel, so inference stops being a memory-bound crawl and starts using the chip the way training does.

01The Mechanism

Start from noise, refine the whole thing

Autoregressive models write one token at a time, left to right; a diffusion model starts from noise and refines every position together over a few passes.

instead of generating images you know left to right one pixel at a time you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end.

— Stefano Ermon, No Priors
Key Insight
The layout of the computation, not the model size, is the design choice here: left-to-right is a scheduling decision baked into autoregressive models, and diffusion simply refuses to make it.
02Hardware Fit

Autoregressive leaves the GPU waiting

Generating one token at a time is memory-bound: the GPU spends its time moving weights and does very little math, so most of the chip sits idle.

you're spending most of your time moving around weights across the memory hierarchy and you're doing very little arithmetic

— Stefano Ermon, No Priors
Key Insight
This reframes the efficiency race. If the ceiling is how fast you can feed weights to the cores, a more parallel generation pattern raises that ceiling without a single new chip, which is why Ermon frames diffusion as an architecture bet rather than a quality bet.
03The Analogy

The transformer moment, but for inference

In 2017 the field moved from RNNs to transformers to make training parallel; Ermon is betting diffusion does the same thing for inference.

a diffusion model is built to have at inference time a workload where you process many tokens at the same time

— Stefano Ermon, No Priors
Key Insight
The analogy carries a warning for incumbents: the last parallelism jump, RNN to transformer, did not preserve the leaders and it rewired the whole stack. Assuming this jump gets absorbed by scale alone assumes it is smaller than the last one.
04Why Inference

Inference is where scaling happens now

Reasoning and RL both push more compute into inference, so a model that is cheaper per token compounds every downstream gain.

If you think about a lot of the advances with reasoning models, a lot of it is scaling test and compute, right?

— Stefano Ermon, No Priors
Key Insight
Inference used to be the cheap part you did once; reasoning and RL turned it into a repeated cost. That flips the usual logic, because an efficiency edge at inference now feeds back into training quality instead of saturating.
05The Wedge

Fast is a one-way ratchet

Speed wins wherever latency matters, by Ermon's estimate 20 to 30 percent of workloads today, and once users feel a fast model they do not go back.

once you get used to a fast model, it's hard to go back. It's kind of like broadband, right?

— Stefano Ermon, No Priors
Key Insight
A one-way ratchet is a moat in disguise: if users never downgrade from fast, the first provider to reach good-enough-and-fast in a latency-critical niche can keep it, even if a rival later matches quality at normal speed.
06Software over Silicon

Custom-chip speed, on a commodity GPU

One voice customer got custom-chip speed from a diffusion model running on ordinary Nvidia GPUs, because the parallelism comes from software.

If you have a diffusion based LLM that it's built to be parallel, it's accelerating at the software level, then you can get the same speed on Nvidia GPUs

— Stefano Ermon, No Priors
Key Insight
The interesting claim is substitution: software parallelism can stand in for exotic hardware. If that holds, specialized inference chips compete not only with each other but with a software recipe that runs on GPUs customers already own.
07New Capability

You can steer it before it finishes

Because diffusion refines coarse-to-fine, you can push generation toward a reward or constraint from the first pass instead of scoring only the finished output.

From the very beginning you know kind of like is this object the kind of thing I want or not and you can steer the generation in the direction provided by an external reward function or some set of constraints.

— Stefano Ermon, No Priors
Key Insight
Steering mid-generation is a different control surface: because a diffusion model holds a rough draft of the whole output early, it can be nudged toward a whole-object reward as it forms rather than scored only once finished. If that matures, it shifts what alignment can mean at inference time, from filtering finished outputs to shaping them as they form.