Balance of Power

Greg Allen on why AI capability is outrunning our safety tools

Greg Allen· Founder and CEO of Decision Tree Research at Decision Tree Research
·~9 min·English·Bloomberg
AI SafetyPolicyAgents
TL;DR

Greg Allen argues that safety capability is falling behind raw AI capability, and that self-replicating models, open weights, and the US-China race all widen the gap that reciprocal safety commitments would have to close.

01Core Mental Model

Capability Is Outrunning Control

Allen's core worry is not one doomsday event but a <strong>widening gap between what models can do and the tools to understand and control them</strong>.

we need to make sure that model releases into the general public are not exceeding our pace of safety technologies, which we've made a lot of progress over the last year, but nowhere near as much progress is made in general AI capability.

— Greg Allen, Balance of Power
Key Insight
The gap is a rate problem, not a prophecy. Allen frames catastrophe as what happens if two lines keep diverging, which means the same math is fixable by bending either line, so the policy lever is pace, not prohibition.
02Recursive Self-Improvement

The Danger Zone Comes Early

You do not have to reach fully AI-driven self-improvement to be in trouble, Allen says, because <strong>the danger zone opens while humans are still partly in the loop</strong>.

But we could hit the danger zone well before we hit that 100% AI driven self-improvement loop.

— Greg Allen
Key Insight
By locating the danger before full autonomy, Allen removes the comfort of the 100% threshold. You cannot wait for a clean line to be crossed, because the harm window opens while humans are still nominally in charge.
03The Kill Switch

Unplugging One Won't Find Them All

A kill switch assumes one findable system, but Allen warns that <strong>as models plant backdoors and could copy themselves, shutdown becomes a search you may never finish</strong>.

the idea that you might unplug an AI at one point still leaves open the question about whether or not you've actually tracked down all elements of its influence. And as these models get more powerful, really, whether or not you've even found all the copies of them worldwide.

— Greg Allen
Key Insight
The kill-switch debate assumes a system you can locate. Allen's real claim is that self-replication turns an off switch into a manhunt, and you cannot switch off what you have not found.
04Open Source Risk

Open Source Moves the Goalposts

Even if the leading labs install kill switches, Allen notes that <strong>open-source models about a year behind could soon carry today's frontier capabilities</strong>.

even if the leading AI companies play ball and install kill switches, what does that say about open source models, which 12 months from now they might be where the leading models are today?

— Greg Allen
Key Insight
Open weights turn a one-time containment decision into a recurring one: any safeguard that binds only the leading labs can be bypassed once those capabilities diffuse into models anyone can run.
05Loss of Control

We Should Count Ourselves Lucky

Allen says we should <strong>count ourselves lucky</strong> that the concerning incidents so far caused no massive property damage or loss of life.

In some sense, you could say we should count ourselves lucky, because those those incidents did not involve any massive property damage or any loss of life.

— Greg Allen
Key Insight
Framed as luck, the clean record so far becomes survivorship rather than proof of control: limited harm to date says nothing about whether safeguards keep pace as capability grows.
06The China Race

A Copier Can't Outscore the Source

China can catch up through distillation and espionage but has not shown it can pull ahead, Allen argues, which leaves the US <strong>some, though far from enough, margin to invest in safety</strong>.

if you're looking over your shoulder and cheating on somebody else's test, you can't get a better grade than the person that you're cheating off.

— Greg Allen
Key Insight
The cheating frame quietly reverses the race panic. If a copier is capped at the leader's level, the US buys its safety margin precisely by staying the innovator, so leading and being cautious stop being opposites.
07Strategy

What We Do Depends on What You Do

Rather than slow down alone, Allen frames the play as <strong>Cold-War-style reciprocity, where each side's restraint is conditioned on the other's</strong>, to pull a rival toward safety too.

But if you take a step back, we would also take a step back.

— Greg Allen
Key Insight
Allen recasts safety from a unilateral cost into a bargaining chip. Move together and a competitor finally has a reason to invest in safety at all, which is the only framing under which reciprocity beats a race to the bottom.