Latent Space

Sean Lie on why yesterday's fast is the new batch mode

Sean Lie· Co-founder and CTO of Cerebras Systems at Cerebras Systems
·~44 min·English·Latent Space
InferenceAI InfrastructureAgentsReasoning
TL;DR

Cerebras co-founder and CTO Sean Lie argues that wafer-scale speed is redefining inference: yesterday's fast is becoming batch mode, speed compounds into smarter agents, and the real frontier is moving to integration beyond the single chip.

01Core Mental Model

Yesterday’s “Fast” Is the New Batch Mode

<strong>The definition of “fast” keeps sliding:</strong> token rates once considered fast now feel like overnight batch jobs, while real-time becomes the new bar.

what used to be batch and offline applications start to become real time

— Sean Lie, Latent Space
Key Insight
The market re-baselines continuously. A chip that is merely competitive on today's tokens per second is already aiming at a target the field is about to abandon; the winning bet is on where the bar lands next, not where it sits now.
02The Flywheel

Speed Compounds Into Intelligence

<strong>Faster tokens are not just a smoother interface:</strong> they buy more agentic loops and more reasoning inside the same time budget, which produces measurably more capable agents.

this flywheel of not just making models you know faster but using that speed to make them more intelligent

— Sean Lie, Latent Space
Key Insight
If speed acts as an intelligence multiplier, then hardware quality and model quality stop being separate axes. A faster chip raises the ceiling on how smart a fixed model can behave, which is why co-designed hardware and models reinforce each other instead of competing.
03The Journey

From “You Guys Are Insane” to Sold Out

<strong>Wafer-scale moved from a technology demo to sold-out capacity:</strong> the first chip only had to prove the idea was possible, and the job now is scaling it.

this is the future and most of them were just like you guys are insane this is not possible

— Sean Lie, Latent Space
Key Insight
The bottleneck has moved from physics to logistics. Once feasibility is proven, the moat is no longer the core idea but megawatts, packaging, and yield at scale, which are far harder for a newcomer to copy than the concept itself.
04Reading the Competition

The 30-Billion-Parameter Tell

<strong>Memory capacity, not raw speed, is the real constraint:</strong> a small-memory SRAM chip can only benchmark a small model because it lacks the memory to hold frontier weights at all.

one of our chips has you know order 100 times more memory than one of their chips right so you have two orders of magnitude difference in scale kind of for free

— Sean Lie, Latent Space
Key Insight
You can read a competitor's memory ceiling from the model size they choose to show. Publishing numbers on a 30-billion-parameter model, when frontier models run into the trillions, signals that the weights do not fit, not that 30 billion was the interesting choice.
05Architecture Vision

Treat the Data Center as One Chip

<strong>Inference is no longer one workload but many:</strong> prefill, decode, KV-cache loading, attention, and expert routing can each get purpose-built hardware inside one disaggregated system.

it's as if you're like thinking about the entire data center as if it's like one computer. It's like one chip that you're trying to figure out

— Sean Lie, Latent Space
Key Insight
The unit of design is shifting from the chip to the cluster. Disaggregation only pays off at hundreds-of-megawatts to gigawatt scale, which is why hyperscale operators, not individual buyers, are the ones who can actually capture this efficiency.
06The Untapped Gain

We’re All Running Models Designed for Nvidia GPUs

<strong>Today’s models are shaped for one specific Nvidia GPU:</strong> even slight architectural adjustments for the target hardware can unlock large additional speedups.

if you start to then open up the possibility of adjusting that model architecture even slightly, you can get massive gains

— Sean Lie, Latent Space
Key Insight
This co-design gain favors whoever controls both the model and the silicon. A hardware vendor forced to run someone else's fixed model can never capture it, which is exactly why the OpenAI and Cerebras pairing is framed as a flywheel rather than a supplier relationship.
07Where the Next Leap Comes From

The Frontier Is Outside the Chip

<strong>No single chip can hold a frontier model,</strong> so the next big gains come from integration across memory, interconnect, and packaging at the package, node, and rack.

the applications, the problems, the models are now so large that you can't just do anything on a single chip. So everything comes down to the integration

— Sean Lie, Latent Space
Key Insight
The differentiator is sliding down the stack toward advanced packaging and 3D memory. That makes memory vendors and DRAM stackers as strategically important as the compute designers, and it reframes the race as a systems-integration contest rather than a transistor one.
08Hard-Won Lesson

The Real Wafer-Scale Boss Was Power

<strong>The hard part was not wiring the wafer but feeding and cooling it:</strong> Cerebras expects 3D DRAM stacking to run into the very same wall.

what turned out to be, you know, the biggest enablers was ultimately how do you power it? How do you cool it

— Sean Lie, Latent Space
Key Insight
There is a general rule here for exotic hardware: the headline challenge is rarely the one that decides success. The mundane physical constraints, power delivery and thermals, are, and they carry over to every next packaging bet, which is why hard-won power and cooling expertise transfers directly to a DRAM-stacking program.