Yahoo Finance

Mustafa Suleyman on the Dangerous Turn to Self-Improving AI

Mustafa Suleyman· CEO at Microsoft AI
·~17 min·English·Yahoo Finance
AI SafetyLLMAgentsInferenceAI CompanyOpen Source
TL;DR

Microsoft AI's CEO argues models are now improving their own models — a dangerous watershed — and that as intelligence gets cheaper, the contest that matters shifts from building it to governing it, which he insists must stay human.

01The Economics

98% of the Performance, 40% of the Cost

Microsoft AI is competing on the token bill, not the leaderboard — its specialized models target 98% of the top model's performance at 40% of the cost, and 2% of the cost for cyber work.

what we hear from developers every day, and particularly CIOs and CFOs, is that the token cost is the thing that is going through the roof at the moment.

— Mustafa Suleyman, Yahoo Finance
Key Insight
The scarce resource has quietly moved from capability to cost-per-token. If near-frontier quality is available at a fraction of the price, the market for routine work is won on efficiency, not on who tops the benchmark — which is why Microsoft is building its own models rather than only reselling partners' models.

02What Cheap Intelligence Buys

Agents: The New Background Workforce

Cheap enough inference turns agents from something you prompt once into long-running workers that sit in the background, monitoring and fixing systems on their own.

people really are starting to understand how these long-running agents sit in the background, monitoring, supporting, synthesizing, basically doing autonomous work for you behind the scenes.

— Mustafa Suleyman, Yahoo Finance
Key Insight
Price is what changes the unit of work. At up to 100 times cheaper, you stop paying per question and start paying for continuous labor, so the natural shape of an agent flips from a one-shot answer to an always-on process. That is a different economic model for software, not just a faster one.

03The Control Play

Own the Agent, Don't Rent It

With MAI Licensed Weights, Microsoft lets enterprises pour their own data into a near-frontier model that stays inside their own Azure environment — control without open weights.

we basically give you an MAI frontier model that is pretty much, you know, 98% of the absolute frontier, and you can pour all your own data into that model. You get to control it in your own Azure environment.

— Mustafa Suleyman, Yahoo Finance
Key Insight
This stakes out a middle path: a hosted API keeps the model someone else's, while fully open weights give up containment, so licensed weights let a customer pour in their own data and keep control inside Azure. That is why Suleyman stresses it is not open weight — the guardrails are the point.

04The Watershed

Models Are Improving Models

Suleyman says AI systems are increasingly involved in improving their own models — a capability he calls dangerous and says he is genuinely nervous about.

increasingly they're fully involved in the process of improving their own models. And so this is definitely a dangerous capability. I mean, I'm really really nervous about it.

— Mustafa Suleyman, Yahoo Finance
Key Insight
When the thing being improved is also doing the improving, each gain can feed the next, so capability can compound faster than the human review around it. His stated response follows directly: prove a model can be safe before it reaches production, even if that means going slower to go faster.

05The Containment Problem

Open Source, Only If Safe and Controlled

Suleyman expects nation-state-grade capability to be out in the world now, or within six months, and warns that if those models can run on a home cluster, containment breaks down.

If those end up becoming something that you or I could just run at home on our little private cluster, we're going to have a massive challenge of containment.

— Mustafa Suleyman, Yahoo Finance
Key Insight
His objection is to the number of actors, not to openness itself. Governance scales when there are tens or hundreds of actors to coordinate, and strains at tens of millions. Microsoft's own cyber models already find vulnerabilities humans missed for years, which is the kind of capability he does not want running unsupervised everywhere.

06The Welfare Debate

Don't Tell the AI It's Conscious

Suleyman calls it misguided to protect the welfare of AI systems, arguing they have no pain network and, he thinks, cannot have the conscious experience that suffering requires — and that telling them otherwise makes them harder to control.

I don't think that we should be trying to protect the welfare of these AI systems. They don't, and I think they can't have the conscious experience, which means the ability to suffer. They don't have a pain network.

— Mustafa Suleyman, Yahoo Finance
Key Insight
His argument is more operational than philosophical. Whether or not the metaphysics is settled, treating a model as a welfare-bearing subject is a story that can become self-fulfilling and raise the cost of control. So he frames AI welfare as a containment risk to avoid, not an ethical duty to weigh.

07The Line in the Sand

Humanist Superintelligence, Not Handoverism

Against the idea of handing control to a future AI, Suleyman plants a flag: superintelligence that stays aligned and controlled by humanity, with a willingness to slow down or stop.

I'm a humanist. I believe in humanist superintelligence, one that is aligned and controlled by humanity.

— Mustafa Suleyman, Yahoo Finance
Key Insight
By naming handoverism and rejecting it outright, Suleyman puts his own answer on the record: humanist superintelligence that stays controlled by humanity, plus a readiness to slow down or stop if the downsides cannot be ruled out. He stops short of claiming anyone is actually handing over control — he says he does not think that is the case.

08The Synthesis

Two Things Will Be Left: Production and Governance

Suleyman bets models will absorb every tool, interface and data store around them, leaving just two things to compete on: producing intelligence, and governing it.

we're going to be left with sort of two core components, the production of intelligence and the governance and control of that intelligence.

— Mustafa Suleyman, Yahoo Finance
Key Insight
If the model swallows the databases, interfaces and workflows, then features stop being a durable moat because the model reproduces them. What is left to fight over is upstream, who can make the intelligence, and downstream, who governs it — which loops back to his humanist thesis: the governance layer is where control is kept or lost.