Mustafa Suleyman on the Dangerous Turn to Self-Improving AI
Microsoft AI's CEO argues models are now improving their own models — a dangerous watershed — and that as intelligence gets cheaper, the contest that matters shifts from building it to governing it, which he insists must stay human.
98% of the Performance, 40% of the Cost
Microsoft AI is competing on the token bill, not the leaderboard — its specialized models target 98% of the top model's performance at 40% of the cost, and 2% of the cost for cyber work.
what we hear from developers every day, and particularly CIOs and CFOs, is that the token cost is the thing that is going through the roof at the moment.
Agents: The New Background Workforce
Cheap enough inference turns agents from something you prompt once into long-running workers that sit in the background, monitoring and fixing systems on their own.
people really are starting to understand how these long-running agents sit in the background, monitoring, supporting, synthesizing, basically doing autonomous work for you behind the scenes.
Own the Agent, Don't Rent It
With MAI Licensed Weights, Microsoft lets enterprises pour their own data into a near-frontier model that stays inside their own Azure environment — control without open weights.
we basically give you an MAI frontier model that is pretty much, you know, 98% of the absolute frontier, and you can pour all your own data into that model. You get to control it in your own Azure environment.
Models Are Improving Models
Suleyman says AI systems are increasingly involved in improving their own models — a capability he calls dangerous and says he is genuinely nervous about.
increasingly they're fully involved in the process of improving their own models. And so this is definitely a dangerous capability. I mean, I'm really really nervous about it.
Open Source, Only If Safe and Controlled
Suleyman expects nation-state-grade capability to be out in the world now, or within six months, and warns that if those models can run on a home cluster, containment breaks down.
If those end up becoming something that you or I could just run at home on our little private cluster, we're going to have a massive challenge of containment.
Don't Tell the AI It's Conscious
Suleyman calls it misguided to protect the welfare of AI systems, arguing they have no pain network and, he thinks, cannot have the conscious experience that suffering requires — and that telling them otherwise makes them harder to control.
I don't think that we should be trying to protect the welfare of these AI systems. They don't, and I think they can't have the conscious experience, which means the ability to suffer. They don't have a pain network.
Humanist Superintelligence, Not Handoverism
Against the idea of handing control to a future AI, Suleyman plants a flag: superintelligence that stays aligned and controlled by humanity, with a willingness to slow down or stop.
I'm a humanist. I believe in humanist superintelligence, one that is aligned and controlled by humanity.
Two Things Will Be Left: Production and Governance
Suleyman bets models will absorb every tool, interface and data store around them, leaving just two things to compete on: producing intelligence, and governing it.
we're going to be left with sort of two core components, the production of intelligence and the governance and control of that intelligence.