a16z Podcast

Simon Mo on why open-source inference became AI's control layer

Simon Mo· Co-founder, lead maintainer of vLLM at Inferact
·~46 min·English·a16z
Open SourceInferenceAI InfrastructureAgents
TL;DR

Simon Mo, lead maintainer of vLLM, argues that open-source inference has quietly become AI's critical infrastructure — the control layer that turns commodity GPUs into intelligence, and where capability parity leaves control over speed, guardrails, and economics as the real contest.

01Core Mental Model

The inference engine is critical infrastructure

vLLM's job is to turn available GPUs into a running endpoint for intelligence, making it the same kind of unglamorous, load-bearing software as a database or an operating system.

That means its job is to turn available GPUs into a running endpoint for intelligence.

— Simon Mo, a16z Podcast
Key Insight
Calling vLLM an operating system is a positioning claim, not modesty: infrastructure is judged on reliability and neutrality, not features. And because hardware vendors run it as their own benchmark, Nvidia, AMD and Google are effectively co-developing against it — which turns the engine into a shared industry standard rather than any one vendor's product.
02Why It's Hard

Serving an LLM broke the old ML playbook

Serving a language model is a scheduling problem, not just a compute problem: requests arrive at unpredictable lengths and finish non-deterministically, so batching and scheduling — not the forward pass — are the hard core.

So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs and it it is a computationally intensive process that will require a lot of engineering

— Simon Mo, a16z Podcast
Key Insight
The reason old ML tooling didn't transfer is that the bottleneck moved from FLOPs to how you pack variable-length, non-deterministic requests onto a GPU. That is why an inference engine is a distinct layer — batching, scheduling, and memory management — rather than a thin wrapper around a single forward pass, and it's exactly the terrain LAV's LLM Serving and Inside vLLM tracks map out.
03The Turning Point

Open source got dragged into the frontier

Open source went from enthusiast curiosity to critical infrastructure when serious app companies realized they couldn't win as a wrapper on someone else's API and needed their own training and inference.

a bunch of smaller companies or like new application companies as they were trying to figure out how do I really build an AI without just being a wrapper on top of open AI. The answer to that question turned out to be open source.

— Matt Bornstein, a16z Podcast
Key Insight
"You can't be a wrapper" is really a margin argument. If your product is just a call to someone else's API, your differentiation — and your gross margin — belongs to the model vendor. Owning inference is how an app company converts a variable, uncontrollable API bill into a fixed infrastructure cost it can shape, which is why open source got embedded deep inside products that never advertise it.
04The Real Driver

Control first, cost later

Control has always been the backbone reason teams run open weights; cost only started to matter recently, which is why owning the model — not saving money — is the durable motivation.

I think like it fluctuate over time. So control matters a lot over the last few years and then costs just start to matter over the last few months.

— Simon Mo, a16z Podcast
Key Insight
Control and cost aren't competing reasons — control is the one that survives. Cost savings can evaporate the moment a frontier open model prices near the closed ones (as Simon notes Kimi K3 does), but owning the SLA doesn't. His voice-agent example makes the point sharp: when a customer is waiting on the phone, response-time control is a product requirement, not a nice-to-have you'd trade away for a cheaper API.
05The Payoff of Control

Two switches or ten speeds

A closed API gives you the vendor's two operating points; running the weights yourself turns speed into a tunable dial with roughly ten tiers, reaching 400–500 tokens per second.

for proprietary model there is uh regular mode and fast mode and that's only the two switch here. But for openweight when you're running it, every provider can offer potentially even 10 different levels of speed

— Simon Mo, a16z Podcast
Key Insight
The "ten levels of speed" is the hidden payoff of open weights: a closed API hands you the vendor's two operating points, while running the weights turns latency into a dial you tune per workload. For agents doing long-running, blocking tasks, that dial — the difference between a model stuck thinking and one that keeps executing — can be what separates a shippable product from an unusable one.
06The Economics

An AI model is not software

Open-source software ran on donated time, but you can't train a frontier model with friends after work — it takes millions to billions in compute, so open-weight licensing exists to fund the next run.

Like like I can't just like go home at night and like train a frontier opensource model with friends for fun. Like we need millions or billions of dollars of computing resources in order to do it.

— Matt Bornstein, a16z Podcast
Key Insight
Treating a model like a drug reframes open-weight licensing from "greed" to "R&D funding." The uncomfortable corollary Simon draws is geopolitical: if Western buyers won't fund a Chinese lab's next run through licensing revenue, someone else — governments — will, an outcome he argues is worse for the open ecosystem than simply paying for the weights.
07Control, Concretely

Own your own guardrail

Closed-model guardrails are arbitrary and hard to enforce, so their false positives block legitimate work — which pushes teams toward open weights, where they can set the guardrail themselves for trusted use cases.

That means they have so much false positive in the guard rails that will have legitimate use cases just being blocked and this is like a evergreen problem even in the social media days

— Simon Mo, a16z Podcast
Key Insight
The guardrail argument inverts the usual safety framing: for Simon's own team, a closed model's caution is the liability. When a false positive can delete a multi-hour job, "safety" becomes a reliability tax, and a guardrail you can tune becomes an operational requirement — the same logic, he notes, that once turned social-media moderation from a feature into an entire platform problem.
08Forward Look

Not even today

Simon sees no real capability gap between open and closed models even now, which pushes the contest downstream to data and reinforcement-learning environments — the part that can't be distilled or copied.

Capability wise, I don't really see a big gap. Not even today

— Simon Mo, a16z Podcast
Key Insight
If capability has converged, moats move downstream. Simon's bet is that the durable advantage is the reinforcement-learning environment a lab builds — which, unlike weights, can't be distilled or copied out. That reframes "who wins" from "who has the best model" to "who can build the best environment to keep improving one," and it's why he treats distillation fears as largely beside the point.