Latent Space

Jason Ma on Robots That Imagine Before They Act

Jason Ma· Co-founder & Chief Scientist at Dyna Robotics
·~80 min·English·Latent Space
RoboticsTrainingInferenceMultimodalAI Company
TL;DR

Dyna Robotics co-founder Jason Ma explains the world action model behind his robots: predict the physical world, then act, and win by owning the body, the data, and the deployment that make that prediction real.

01Core Mental Model

Imagine the Future, Then Act

A world action model predicts the next moments of video and the robot action together, so the robot decides what to do by first imagining how the world will unfold.

if you think about how humans act in the physical world in our brain, we are implicitly thinking about how the world will evolve based on our actions

— Jason Ma, Latent Space
Key Insight
The framing quietly turns robot control into a prediction problem, arguably the same bet that made language models work. If predicting the next word scaled, the wager here is that predicting the next frame of reality can too.
02Architecture

Why Diffusion, Not Next-Token

Robots use a diffusion transformer rather than a next-token model because actions are continuous and must be smooth and precise, and flow matching cuts the refinement from dozens of steps to about four.

you need the transformer aspect because transformer is the thing that can just ingest the most amount of data can learn then diffusion helps you to refine the outcome to get to the smooth precise fine grain motion you need to actually execute on the robot

— Jason Ma, Latent Space
Key Insight
Choosing diffusion over next-token is a claim about where robotics is hard: precision in continuous space, not juggling discrete symbols. That is the opposite of where a language model spends its effort.
03The Data Problem

Robotics Has No Internet

Unlike language models, robotics has no ready-made internet-scale corpus, so Dyna builds its data from abundant first-person human video for a broad prior plus scarce robot data for dexterity.

data is the most important thing in robotics today because unlike language model, there is no internet, right?

— Jason Ma, Latent Space
Key Insight
No internet is the real moat. Whoever assembles the missing robotics corpus, not whoever has the cleverest architecture, likely wins, which is why the Dyna-2 blog post ends with a request for data partners.
04Scaling

The Self-Fulfilling Flywheel

Because the model can generate video as well as actions, it can produce its own training data and synthetic scenes, turning data generation into a flywheel that spins faster the better the model gets.

the model not only generates action, can also generate videos, the model can also generate its own training data. So that becomes a you know self-fulfilling like flywheel

— Jason Ma, Latent Space
Key Insight
If the model can be trained on the rollouts it imagines, data starts to look less like a fixed input and more like something renewable, a robotics echo of the self-generated feedback that powered reinforcement learning from human feedback.
05Deployment Constraint

Gravity Does Not Wait

A robot cannot pause mid-motion the way a coding agent can, so inference has to run on-device in a fraction of a second, which is why Dyna trains a large model and distills it down to a small one for the robot.

the robot needs to act like it can't just hold a knife and not do anything for three second that's going to freak people out right

— Jason Ma, Latent Space
Key Insight
Latency looks like the binding constraint that pushes the whole stack in-house. A cloud API can be accurate yet still struggle to guarantee a physics deadline, so the serving model gets pushed onto the robot itself.
06Full Stack

Hardware Carries Intelligence

Dyna builds in-house only the parts that carry the intelligence, like actuators and a human-matched body, and buys mature commodity parts, landing on a semi-humanoid that puts a humanoid torso on wheels.

eventually you know it's hardware that carries intelligence. Intelligence is the extremely important thing. We want to make sure the hardware serves for that

— Jason Ma, Latent Space
Key Insight
Matching the robot's proportions to a human's is a data decision disguised as a hardware one. The body is being shaped so that human training video transfers, rather than forcing the data to fit an arbitrary machine.
07Go To Market

A Pilot Does Not Count

The hard milestone is not one working pilot but a customer validating enough value to deploy across many sites, and a single generalist model that adapts to a new task from as little as tens of minutes of demonstrations beats a specialized machine on deformable objects.

what's really hard is not putting your robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they want to deploy your robots across many of their sites

— Jason Ma, Latent Space
Key Insight
A single pilot proves the robot can work once; a signed expansion proves a customer will pay for it repeatedly. Only the second one compounds into a business, which is why he draws the line there.
08First Principles

The Bitter Lesson for Robots

Scaling will solve robotics the way it solved language and vision, but the uniquely hard part is choosing which sensors and data to scale, so the recipe is a simple algorithm predicting the raw world rather than hand-built priors.

just keep the algorithm simpler and scale the right kind of data then generalization better behavior will emerge

— Jason Ma, Latent Space
Key Insight
The deepest bet is that robotics is not special: the scaling that solved language and vision can carry over to manipulation, once someone does the unglamorous work of deciding which sensors belong on the robot.