Chris Lattner on why one AI stack can span every chip
Chris Lattner explains why AI's hardware is fragmenting into specialized chips, how one software stack can span them by unifying what is common and specializing only the hot path, why selling Modular to Qualcomm was what finally let the platform go open source, and where he still keeps a human in the loop as agents take over the code.
No single chip fits the whole workload
The inference workload is not one job but three — prefill, decode, and speculative decoding — and each one wants a different kind of silicon, so no single chip wins the whole thing.
different parts of the workload can be run by different chips. And so you get some chips that are good at what's called the prefill phase of compute, some that are great at the decode phase, some that are good at what's called speculative decoding.
Default to compatible, specialize for alpha
Lattner's stack unifies everything that is architecture-independent — batching, the KV cache, speculative-decode logic — and specializes only the innermost matmul, so code defaults to compatible but still exploits each chip.
instead of everything defaulting to being different you can default to being compatible but still specialize where you get alpha out of taking advantage of the chip.
Start at the silicon, not on top of it
Instead of bolting a compatibility layer onto decades-old stacks, Modular started at the silicon and built the whole tower up, so shared math primitives compose into shared attention blocks and shared serving.
Let's build that programming language. Let's build the programming model. Let's build the AI kernels for doing the math. Let's build the serving architecture. Let's build this entire stack from the bottom up.
A startup can't afford to make it open
A startup has to protect its IP and find a way to monetize, so it cannot give the platform away; an owner like Qualcomm that profits from the silicon can make the software open.
And as a startup, we couldn't really do that. We had to protect the IP. We had to figure out how to monetize it.
From the data center to the earbud
A startup only makes money in the cloud, so it only cares about the data center; Qualcomm's reach from server to earbud gives the same stack a reason to scale down to every power envelope.
you can only make money in cloud. And so, as a startup, we're only incentivized to care about the data center.
Mojo: the Python family, minus the performance tax
People leave Python reluctantly, only for performance — so Mojo stays inside the Python family and fixes the speed, distribution, and packaging that force them out.
all the kind of people that leave Python do it very reluctantly because they have to for performance reasons not because they don't enjoy the language.
Compounding S-curves, not a strict exponential
AI does not move on one smooth exponential; it moves on compounding S-curves — pre-training rose and flattened, RL layered on a new curve, and the next one stacks on top of that.
And the thing that propels AI, in my opinion, isn't strict exponentials. What it is is it's compounding S-curves.
A human on the architecture, agents on the spec
Lattner splits the work by whether a clean spec exists — a human stays in the loop on architecture that needs a 6-to-12-month view, while agents are turned loose on anything with a behavioral spec.
if you're working on the architecture a human has to be in the loop because you need somebody that's thinking about not just what are we doing today but what are we doing 6 months or 12 months from now