Latent Space

Akshat Bubna on why agent experience is the new developer experience

Akshat Bubna· CTO of Modal at Modal
·~59 min·English·Latent Space
AI InfrastructureAgentsInferenceTraining
TL;DR

Modal's CTO Akshat Bubna on why the infrastructure that made developers productive is exactly what AI agents need, why a single RL run can fan out to 100,000 sandboxes, and how a capital-light super-cloud with no data centers plans to serve it.

01Core Mental Model

DX Is Now AX

Modal rebuilt its SDK around agent experience, betting that the ergonomics which made humans productive make agents productive too.

why would you have an agent read through hundreds of Kubernetes files and like write YAML that's not even typed when it can basically make a couple changes in a decorator

— Akshat Bubna, Latent Space
Key Insight
The pitch quietly reframes lock-in fear: an agent that edits one typed decorator instead of hundreds of untyped YAML files fails less and self-corrects faster, so 'good agent experience' is really just a lower error surface for a non-human operator.
02Origin Story

Sandboxes, Two Years Early

Modal shipped agent sandboxes in May 2023 and watched them sit quiet for two years before the agent wave made them essential.

we built sandboxes in May of 2023 before anyone was even knew this was going to be a thing.

— Akshat Bubna, Latent Space
Key Insight
Building the right primitive before the market exists looks like dead weight, because the first demo, an agent looping on itself, diverged after roughly ten iterations on models that weren't post-trained for it, but it means the capacity, APIs, and muscle memory are already there when demand finally arrives.
03The Technical Heart

RL Is Insanely Bursty

A single reinforcement-learning run can fan out to 100,000 sandboxes at once, making elastic scale-from-zero the workload's hardest requirement.

RL is insanely bursty.

— Akshat Bubna, Latent Space
Key Insight
Reinforcement learning inverts the classic serving problem: instead of steady traffic you get violent spikes, so the moat isn't raw GPU access but scaling from zero to around 1,500 GPUs in a single region fast, which is why Modal snapshots GPU state to make the next cold start cheap.
04Inference, Demystified

Accept Length, Not Faster Kernels

The real inference speedup comes from a draft model whose guesses survive verification, not from shaving microseconds off kernels.

people talk a lot about we made these kernels faster and whatnot but improving kernel only give you like a few percentage points of improvement and increasing accept length literally is a multiplicative decrease in like 2 to 4x without much head-on performance.

— Akshat Bubna, Latent Space
Key Insight
Kernel work is additive and capped; accept length is multiplicative because a longer accepted block turns memory-bandwidth-bound single-token decoding into compute-efficient batched verification, the same roofline logic that governs the rest of GPU serving.
05The Business Model

A Super-Cloud With No Data Centers

Modal owns no hardware, running across 17 cloud providers behind its own reliability layer so a dead GPU never reaches the user.

being capital light and focusing on the software helps us move really fast.

— Akshat Bubna, Latent Space
Key Insight
Staying capital-light while other companies build the data centers is a deliberate wedge: the reliability layer that hides a GPU 'falling off the bus' lets Modal safely tap cheaper, flakier capacity across 17 providers that a customer couldn't reliably use on their own.
06The Macro Shift

The Inference Inflection

As agents constantly block on CPU work, the old 8-to-1 GPU-to-CPU ratio is collapsing toward 1-to-1.

it used to be like let's call 8:1 GPU to CPU and now it's more like 1:1

— Latent Space
Key Insight
When the bottleneck ping-pongs between GPU and CPU on every agent step, keeping the two far apart wastes both, which is the physical reason 'collocate everything' has turned from a nice-to-have into an infrastructure requirement.
07Where He Pushes Back

Agents Need Hard Boundaries

Bubna is skeptical of LLM-mediated permissions at the sandbox level, insisting on hard guardrails with softer ones layered on top.

I'm skeptical of LM mediated permission for stuff that is at the sandbox level because you do want hard boundaries.

— Akshat Bubna, Latent Space
Key Insight
It's a security stance dressed as an infra opinion: an LLM asked to police its own egress can be talked out of the rule, so the sandbox enforces the exfiltration limit in the kernel and leaves the model only the soft, revocable decisions.