Training Data

Rich Sutton & Khurram Javed on why AI must never stop learning

Rich Sutton & Khurram Javed· Co-founders at Oak
·~54 min·English·Sequoia Capital
LLMAgentsTrainingReasoningAI Company
TL;DR

Reinforcement-learning pioneer Rich Sutton and his Oak co-founder Khurram Javed argue that today's LLMs stop learning the moment training ends, and that real intelligence requires continual learning from an infinitely complex world.

01Core Mental Model

Not Him, the Field Is Weird

<strong>All learning is continual</strong> — treating it as a special phase you can switch off is the anomaly, not the other way around.

I'm not weird. The field is weird. The field they need to call it continual learning. It's just learning.

— Rich Sutton, Training Data
Key Insight
By reframing 'continual learning' as just 'learning,' Sutton shifts the burden of proof: the industry, not he, is the one making the strange assumption that a mind can ever be finished.
02The Bitter Lesson

The Bitter Lesson in 26 Words

<strong>Bet on methods that scale with computation</strong> — search and learning — not on hand-coded human knowledge that stops paying off.

don't be distracted by human knowledge as AI traditionally has been many times. Instead focus on learning methods that will scale with computation like search and like learning.

— Rich Sutton, Training Data
Key Insight
The subtle misreading Sutton warns against is treating the Bitter Lesson as anti-algorithm; his actual target is human priors that do not scale, not clever methods that do.
03LLMs, Judged

Proof and Casualty at Once

LLMs are <strong>both the Bitter Lesson's best proof and its coming casualty</strong>: they scaled by drinking the internet, then hit the ceiling of a finite internet.

And uh the world is big and the world is massively bigger than everything we stored on the internet.

— Rich Sutton, Training Data
Key Insight
If the internet is a one-time fossil deposit, then every lab racing on synthetic data is refining the same finite reserve, which is why Sutton calls that race a mistake rather than a solution.
04Big World Hypothesis

The World Is Too Big to Simulate

<strong>The world is infinitely more complex than any model of it</strong>, so no amount of synthetic data can replace learning from real experience.

The world is infinitely complex, and any simulation of it is like microscopic.

— Rich Sutton, Training Data
Key Insight
Because the world contains other minds as complex as your own, no agent can ever hold a complete model of it, so severe approximation, and therefore endless re-learning, is not a flaw to engineer away but the permanent condition of intelligence.
05The Core Gap

Frozen the Moment Training Ends

A deployed LLM <strong>never changes its weights</strong>, so it cannot learn anything new while you use it — the opposite of a plastic brain.

Their weights never change.

— Rich Sutton, Training Data
Key Insight
In-context memory and personalization hide the gap: the model appears to adapt, but the new knowledge lives in the prompt, not the weights, so nothing durable is actually learned.
06The Algorithmic Fix

Catastrophic Forgetting Is Curable

Catastrophic forgetting is <strong>an algorithmic gap, not a law of nature</strong>: per-weight step sizes plus generate-and-test let a network keep learning without erasing what it knew.

Most use cases you have a single stream of data. And then if you apply it to the naive thing, it just completely destroys your prior knowledge in a very um destructive way.

— Khurram Javed, Training Data
Key Insight
Randomness behaves like a consumable resource: standard backprop spends its initial noise and then ossifies, so the fix is less a new objective than a way to keep replenishing plasticity.
07The Plan

Step 2 Unlocks Everything

<strong>Continual deep learning is step two of the Alberta Plan</strong> because once a system can keep updating its world model, abstraction and planning follow.

And we think that one is like almost the most important because it unlocks everything else. If you could do continual deep learning, you could then continually update your model of the world.

— Rich Sutton, Training Data
Key Insight
Framing continual deep learning as the unlock explains why Sutton is not merely improving LLMs: without it, better abstraction and planning have no stable ground to stand on.
08Right-Sizing LLMs

A Quarter of Intelligence

Language is <strong>only about a quarter of intelligence</strong>, so LLMs are a genuine breakthrough that was mistaken for the whole of AI.

All of intelligence is not fluid, capable use of language. There's so much more. It's an important part. You know, it's like 20% or a quarter of intelligence.

— Rich Sutton, Training Data
Key Insight
Calling language a quarter of intelligence is both praise and a boundary: it credits the breakthrough while denying it the extrapolation to all of intelligence that much of the field assumes.