The TWIML AI Podcast

Justin Johnson on What a 'World Model' Actually Is

Justin Johnson· Co-founder at World Labs
·~66 min·English·TWIML
MultimodalReasoningTrainingAI Company
TL;DR

World Labs co-founder Justin Johnson says the term world model covers three distinct ideas, and argues spatial AI can use either explicit 3D representations or implicit models learned from data, rather than text prediction alone.

01Core Mental Model

Beyond the terminal

World models are the bet that AI which understands, generates, and acts in space will not simply fall out of making language models bigger.

we want to build systems that are not just stuck in a terminal or stuck as a virtual agent

— Justin Johnson, The TWIML AI Podcast
Key Insight
Johnson is not saying language models are wrong. He is saying whole classes of ability, such as acting in space and simulating environments, do not emerge from scaling text prediction, so a second modeling paradigm has to grow alongside LLMs rather than replace them.
02The Definition Problem

One term, three systems

The field has no shared definition of a world model, so three quite different systems all get called by the same name.

There isn't a clear definition of world models that everyone in the field agrees on. And I think that's causing part of the confusion

— Justin Johnson, The TWIML AI Podcast
Key Insight
The confusion is not academic nitpicking. When one label covers an implicit-knowledge model, a reinforcement-learning predictor, and a scene generator, benchmarks and research claims get compared across things that are not comparable. Johnson's proposed fix is cleaner terminology, not a new algorithm.
03The Formalism

The POMDP loop

Underneath the competing systems sits one loop: an agent takes actions, the world state changes, and the agent gets back a thin observation of that state.

you sort of partition the whole universe into agent which moves around and does stuff and then world which has stuff done to it or by the agent

— Justin Johnson, The TWIML AI Podcast
Key Insight
POMDP comes from the reinforcement-learning literature, but Johnson deliberately drops the reward term and keeps only the agent-state-observation structure. That is why the same long-standing formalism now describes video generators and robots whose training need not involve reinforcement learning at all.
04A Crucial Distinction

Reconstruction is not a world model

Fitting a point cloud to a thousand photos rebuilds one scene but learns nothing transferable; a world model is the trained prior that can complete scenes it has never seen.

There's no generalizable knowledge here and that's actually very different from what we're doing in Marble

— Justin Johnson, The TWIML AI Podcast
Key Insight
The phrase carries its weight on the word model. The value is not in the Gaussian-splat representation, which both methods share, but in the learned prior that can guess the chairs behind a blackboard it never photographed. A flawless reconstruction of one room teaches a system nothing about any other room.
05The Technical Key

Splats, not triangles

Gaussian splats won over triangle meshes because neural networks learn by gradients, and a splat's soft edges change smoothly with their parameters while a triangle's edges jump.

if I've got a triangle here and I move it a little bit all of a sudden something that became that was invisible now becomes visible

— Justin Johnson, The TWIML AI Podcast
Key Insight
A decades-old graphics primitive lost not on picture quality but on a training requirement: gradient descent needs the output to change a little when the input changes a little. Hard triangle edges create sudden jumps that break that, so splats, which are smooth and semi-transparent, plug into neural networks cleanly.
06The Engineering Tradeoff

Consistency two ways

A world can stay consistent either by construction, through explicit 3D like splats, or by scale, when a big model trained on enough data simply learns to be consistent.

it's more of an engineering question of what are the design constraints of the problem facing me right now and less a philosophical divide for me

— Justin Johnson, The TWIML AI Podcast
Key Insight
Treating consistency as an engineering choice rather than a principled one is the key move. The right representation then depends on your compute budget, not on which method is more elegant, which is why World Labs explores both directions: Marble as an explicit-3D product and RTFM as an implicit, pixels-only research system.
07The Framework

Renderer, planner, simulator

Most systems called world models emphasize one part of the loop: observations make a renderer, actions make a planner, and states make a simulator.

it's not that these people are all building totally different things. They're just focusing on different parts of this fundamental POMDP loop

— Justin Johnson, The TWIML AI Podcast
Key Insight
The taxonomy reveals that a video generator and a robot policy are mirror images: one turns actions into observations, the other turns observations into actions. Framed that way, progress on either side should transfer to the other, since they solve dual halves of the same loop.
08Where It's Going

One unified model

Johnson expects specialized renderers, planners, and simulators to converge into one shared model with different input and output modes.

We're going to have like these giant unified world models that have maybe different input heads, different output heads

— Justin Johnson, The TWIML AI Podcast
Key Insight
If one trunk can act as renderer, planner, or simulator by swapping heads, then a world model becomes a mode you invoke rather than a product category, the way one network does classification or regression from a shared representation. Johnson's clearest architectural bottleneck is context length, since a single world can demand hundreds of thousands to tens of millions of tokens.