The TWIML AI Podcast

Greg Burnham on the straight line hiding AI's jagged frontier

Greg Burnham· Capabilities Research Lead at Epoch AI
·~67 min·English·TWIML
ReasoningLLMTraining
TL;DR

Epoch AI's capabilities-research lead on why a decade of AI benchmarks, stitched together, traces one stubbornly straight line of progress, and what that line hides: a jagged frontier where AI cracks century-old math problems but hasn't yet shown it can invent a genuinely new idea or learn a board game on the fly.

01Core Mental Model

Past the Exam Phase

AI has graduated from the exam room to the job site: the tests we leaned on through early 2025 came with answer keys, and the open-ended problems Epoch now cares about don't.

We're past the academic exam phase of understanding AI capabilities and for that we need high quality benchmarking, high quality evaluations.

— Greg Burnham, The TWIML AI Podcast
Key Insight
The reframe moves the hard part of AI from the model to the ruler. Once the answer key disappears, how good is this system stops being a score you can read off and becomes a research problem in its own right, which is precisely the job Epoch is staking out.
02The Trajectory

Grade-School Math to a Millennium Prize in Two Years

In roughly two years, AI went from struggling with grade-school word problems to solving a Millennium Prize problem, a climb so fast it is easy to stop noticing it.

we're all living through this trajectory, but to step back and see, you know, how much has happened in the past couple of years is pretty pretty nuts.

— Greg Burnham, The TWIML AI Podcast
Key Insight
The danger in a smooth, fast climb is that each step feels normal in the moment. Burnham's timeline is really an argument against calibrating your expectations to last quarter: the same slope that produced grade-school math two years ago just produced a Millennium Prize result.
03How Progress Is Measured

Stitching Sigmoids Into a Straight Line

Every benchmark eventually saturates at 100%, but Epoch stitches those S-curves together and the composite index climbs in a stubbornly straight line.

We've just got a statistical methodology for stitching sigmoids together and what comes out is a linear function.

— Greg Burnham, The TWIML AI Podcast
Key Insight
The straight line is reassuring and misleading at once. It is partly an artifact of how saturating curves average out, so reading it as a law of nature risks mistaking a property of the measurement for a property of the technology.
04The Open Question

Shallow Learning or Deep Generalization?

Every benchmark rises at once, and the open question is why: labs hand-feeding training data for each skill (shallow), or the model generalizing from first principles (deep).

Right now there's a huge industry of collecting all this rich training data and it's really unclear how much generalization you're getting from you know in this deep way.

— Greg Burnham, The TWIML AI Podcast
Key Insight
This is the real stakes question hiding behind a statistical curiosity. If progress is mostly shallow, AI stays roughly as predictable as its training data; if it is deep, capabilities could appear in areas no one trained for, and bounding what the system can do gets much harder.
05Where Discontinuities Hide

Everything's on Trend, But the Trend Is Crazy

Model releases keep landing right on the trend rather than above it, so the one thing that could cause a real discontinuity is AI getting good at improving AI.

everything just about is on trend. Fable uh Astra whatever is on trend but the trend is crazy.

— Greg Burnham, The TWIML AI Podcast
Key Insight
Epoch's worldview pins progress to physical inputs, compute and data, which is why they expect no sudden jumps from the model alone. The trip wire exists because that logic has one escape hatch: an AI that improves AI routes the feedback loop around the physical world entirely.
06The Jagged Frontier

What an AI-Shaped Problem Looks Like

AI's strengths are jagged: it has every paper memorized and infinite patience to brute-force a swarm of attempts, but we still rarely see it build a theory from the ground up.

Maybe I could say AI has two big advantages right now. One is it knows everything like it knows all the prior work. It's got the literature memorized.

— Greg Burnham, The TWIML AI Podcast
Key Insight
The shape of what AI is good at tells you what to automate first. Tasks that reward memorized context and brute-force persistence are already in reach; tasks that require building new conceptual scaffolding are where humans still hold the frontier.
07The Missing Capability

The One Idea AI Hasn't Had Yet

AI is excellent at recombining ideas that already exist, which is most of human math work, but inventing a genuinely new concept is the one thing we haven't yet seen it do.

that's the sort of thing we don't yet see AI doing. It's also the sort of thing that's rarer for humans to do.

— Greg Burnham, The TWIML AI Podcast
Key Insight
Burnham is drawing a line between competence and creativity that most benchmarks blur. The fact that new ideas are rare even for humans suggests this may be the last capability to fall, not a sign the whole enterprise is stuck.
08Can AI Take Your Job?

Today's AI Doesn't Learn This Board Game by Replaying It

On Epoch's board-game test, today's AI arrives good or bad out of the box and shows little gain from replaying it; for now it improves between generations, not by learning on the fly.

they can't create the strategy guide for themselves which is like pretty interesting.

— Greg Burnham, The TWIML AI Podcast
Key Insight
On-the-fly learning may be the hinge between a useful tool and an autonomous worker. Burnham's hunch that it could be a harness problem rather than a model problem is the quiet risk: a capability that looks absent in the lab could switch on the moment the scaffolding around the model improves.