Latent Space

Philip Kiely & Ali Taha on why inference still ships 10x, not basis points

Philip Kiely and Ali Taha· Inference engineers at Baseten at Baseten
·~103 min·English·Latent Space
InferenceAI InfrastructureGPULLMOpen Source
TL;DR

Baseten's Philip Kiely and Ali Taha open up the inference stack — cache-aware routing, error-canceling quantization, speculators, and cross-cluster race conditions — to argue inference is a young enough field to still win in multiples, and one that's collapsing into training to build models that optimize themselves.

01How Inference Works

Have You Sent Me This Query Before?

A modern inference stack's first question isn't which model to run but whether it can skip work it already did, so a long query flows through cache-aware routing, disaggregated prefill and decode, and a traffic-tuned speculator before a single new token is computed.

the first thing that I'm going to ask is, have you sent me this query before or at least part of it? Um, and I really hope you have because it's going to be a lot easier for me and a lot cheaper for you.

— Philip Kiely, Latent Space
Key Insight
The whole economics of serving hinges on avoidance: the cheapest token is the one you never recompute. That's why prefix caching and prefill/decode disaggregation come before raw kernel speed — they decide how much of the 200K-token bill you can simply refuse to pay.
02The Hidden Work

Make a Token, or Ship an API

Getting a brand-new open model to emit one plausible token is easy and mostly free from the open-source stack, but turning it into a service you can put SLAs behind means requantizing to NVFP4, training a speculator on the real weights, porting novel architecture, and patching failures users find in the wild.

there's a difference between support the model as in like I can make a token out of this model and support a model as in I have a production ready API from this model

— Philip Kiely, Latent Space
Key Insight
The launch-day race to tweet “we support model X” rewards the easy part and hides the hard one. The moat isn't day-zero support — it's the first weeks, when a popular model is being beaten on by real traffic and you're still discovering and patching failure modes like mode collapse that no pre-launch test surfaced.
03Open Source

Kimi Vision, GLM Weights, DeepSeek Attention

Because open weights ship as separable parts, the team bolted a frozen Kimi vision encoder onto frozen GLM weights — whose architecture already borrows DeepSeek's sparse attention — by training only a small projector between them, ending up with a model no single lab shipped.

all of a sudden you have Kimmy vision, GLM weights and DeepSeek attention all in one model

— Philip Kiely, Latent Space
Key Insight
This is composability the closed labs structurally can't match — you can only remix parts you're allowed to open and freeze. The move is deliberately conservative: touching the GLM weights risks making the model dumber elsewhere, so only the tiny projector is trained, adding vision without disturbing the intelligence already paid for.
04Quantization

One Layer Screws Right, One Screws Left

Quantization is genuinely lossy, but the loss isn't monotonic: if you choose which layers to compress so their rounding errors point in opposite directions, they cancel, and a model that is more quantized can track the original's output distribution more faithfully than one that is less quantized.

it is possible that the model in which I quantize more information is going to perform better because the quantization errors have canceled out

— Ali Taha, Latent Space
Key Insight
Scoring on benchmarks alone would miss this — the win only shows up when you measure KL divergence between the quantized and full-precision logit distributions. Treating quantization as a search for error-canceling layer sets, rather than a fixed penalty, is how they claim ~20% more of the model in NVFP4 than a stock vendor quant, buying throughput and fidelity at once.
05The Frontier

20%, 100%, 200% — Not Basis Points

Inference is an unoptimized enough field that wins still come in multiples rather than fractions of a percent, and stacking a few roughly-2x optimizations — quantization, a speculator, disaggregation, better kernels — compounds a 30-40 tokens/sec baseline into hundreds.

We're trying to make things substantially faster, not just go from like 70 to 90

— Philip Kiely, Latent Space
Key Insight
The 1970s-finance analogy is also a forecast: the multiples are a sign of immaturity, and they will shrink to basis points as the field is solved. The strategic read is that the returns to inference engineering are highest right now, and the window where an 8x is on the table is the thing that's closing.
06Edge vs. Data Center

Less Dumb vs. Less Slow

Local and data-center inference are the same craft from opposite constraints: on a laptop you fight to fit the model at all, so the goal becomes making it less dumb, while in the data center you start from loading the model and the goal becomes making it less slow.

With local AI, it's how do I fit this model onto my hardware and then make it less dumb? And with data center inference, it's how do I load this model and then make it less slow.

— Philip Kiely, Latent Space
Key Insight
The constraint dictates the toolkit: memory-starved edge work pioneers aggressive dynamic quantization and pruning, while bandwidth-rich data centers lean on tensor parallelism and disaggregation. The interesting arbitrage is that the edge's compression tricks keep leaking upstream — the two ends of the spectrum quietly teach each other.
07Debugging Reality

Software, Weights, or Hardware?

The hardest serving bugs are nondeterministic and cross-layer: a model that collapses into repeating one token on one cluster runs clean on another because the second cluster's slower interconnect exposes a race condition in a kernel, so the same symptom can live in the software, the weights, or the hardware.

if you host it on a different cluster, you will. And the reason is the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster.

— Ali Taha, Latent Space
Key Insight
This is why “we quantized it wrong” is usually the wrong diagnosis — identical weights on a different engine don't reproduce the bug, which points at software and timing, not the model. Serving at scale turns model quality into a distributed-systems problem, where the fix is sometimes just refusing to schedule a model on the cluster whose wiring surfaces the race.
08Where It's Going

The Model Writes Its Own Kernels

Training and inference are collapsing into one loop: the team plugs a strong coding model into their harness to profile the engine, find the bottleneck kernels, and write faster ones, so some of the GPU kernels serving that model in production were written by the model itself.

you are doing inference, learning from the inference

— Philip Kiely, Latent Space
Key Insight
The candid caveat is that today's models still reward-hack and make poor engineering decisions, so the loop needs supervision — this is assisted self-optimization, not autonomy. But the direction is unmistakable: as inference traces feed post-training and models tune their own serving, the line between “building the model” and “running the model” stops being a clean handoff.