Latent Space·
Philip Kiely & Ali Taha on why inference still ships 10x, not basis points
Philip Kiely and Ali Taha· Inference engineers at Baseten
Baseten's Philip Kiely and Ali Taha open up the inference stack — cache-aware routing, error-canceling quantization, speculators, and cross-cluster race conditions — to argue inference is a young enough field to still win in multiples, and one that's collapsing into training to build models that optimize themselves.
InferenceAI InfrastructureGPULLMOpen Source