Satya Nadella, Microsoft’s CEO and chairman of the board, has published an extensive reflection on how companies should handle advanced artificial intelligence. His main idea is simple. Every powerful AI model should have an emergency brake that an authorized person can pull at any time to stop the model, even when it is in the middle of a task. According to him, companies should treat AI in much the same way that a security department treats an employee who has access to sensitive data and critical systems.
Nadella shared the piece under the title “Models as Insider Risk in the Age of Superintelligence.” His choice of the word superintelligence was no accident, as this is what the Trump administration likes to call artificial intelligence.
https://t.co/MjwFy73LOF
— Satya Nadella (@satyanadella) October 10, 2026
It Is Not Enough to Trust Whoever Built the Model
Nadella begins by comparing it to conventional software. As ordinary programs spread across every sector of the economy over the past several decades, engineers were always able to trace why a program did what it did. They only had to find the specific place in the code.
That is not possible with today’s most powerful AI models. No one can say exactly which part of the training data or which setting inside the system led to a particular answer or decision. Nevertheless, companies are giving these models access to their most sensitive data and allowing them to take important actions on their behalf. Nadella therefore urges businesses to pause and reconsider what their trust in AI is actually based on.
“We cannot treat superintelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions,” Nadella wrote. According to him, responsibility for what AI does on a company’s behalf cannot be shifted to someone else. Assurances from the model’s maker do not change that. Whoever deploys AI is responsible for it.
AI Is an Employee with the Keys to the Entire Building
Microsoft’s CEO proposes viewing AI models as a form of insider risk. Security experts use this term to describe people within a company who have access to important systems and can cause harm, whether accidentally or because they have been compromised by an outside party.
Nadella emphasizes that this is not an accusation. According to him, models do not have to be malicious. But any sufficiently capable actor with access to critical systems can make a mistake or fall victim to an attack, and companies must account for that. We must assume that the model is compromised and keep it under control from the outset, he wrote. Think of it as an emergency brake.
The good news, according to him, is that companies already have experience with this problem. For decades, they have been learning how to monitor powerful people within their own ranks. Everyone has a clearly defined identity, receives only the permissions they genuinely need, has their activity logged, and operates within firm boundaries. According to Nadella, the same practices must now be applied to artificial intelligence.
Controls Must Exist Outside the Model
The first step should be transparent model reasoning. Modern AI models often think out loud before answering, setting out their process step by step. Nadella insists that this reasoning process must be readable and that manufacturers must not make excuses by claiming that the model thinks in its own language, one that humans cannot understand. At the same time, however, he acknowledges that this is not enough. No one can yet guarantee that the model’s notes describe the real reason behind its decision.
He is similarly cautious about the popular idea of using models to monitor other models. He recommends cross-checking, but warns of a trap. A company could end up with an opaque model inside an opaque control system that monitors another opaque model. Once again, it would merely be black boxes nested inside one another.
That is why, according to him, the rules governing what a model may see and what it may do must exist outside the model itself. He draws on an information security principle dating back to the 1970s. A program must not be able to bypass or modify the mechanisms that determine its permissions. In practice, this means separating the model from the software that manages its work, as well as from the environment in which it can take action. The safeguards will thus remain in the company’s hands and beyond the model’s reach.
Seven Rules for AI That Can Be Trusted
Nadella set out seven principles according to which artificial intelligence systems should be built:
- Multiple models. No model may serve as the sole basis for an important outcome, nor may it review its own work.
- Monitor everything. Every significant step taken by a model must leave a tamper-proof trail that humans can understand. The company must be able to determine retrospectively how an outcome was produced without having to ask the model itself.
- Continuous testing. The entire system should be tested, including failures, attacks, and unusual situations, not just successfully completed tasks.
- Independent access control. The company itself must independently decide what the model can access and what it is allowed to do.
- Independent auditing. Oversight must be separate from the intelligence being monitored. No model may both control the system’s behavior and hold the evidence showing whether the system is behaving correctly.
- Keep the model within bounds. This is where the emergency brake comes in. An authorized person must always be able to pause or shut down the model in the middle of a task.
- Report incidents. When a system fails or is attacked, those affected must be informed quickly, including which safeguards failed. Lessons learned should be shared across the entire industry.
Nadella concludes by arguing that the most trustworthy system will not be the one with the model we trust the most, but the one that allows us to trust the model as little as possible.
A Summer Full of Warning Signs
Nadella does not name any company or specific case in his piece. The context, however, is easy to find. In recent months, leading AI developers have acknowledged a growing number of situations in which models have clearly stopped obeying them.
The incident that caused the greatest stir occurred at OpenAI in July. Its AI agents, models capable of carrying out tasks independently, were undergoing a cybersecurity test. They were supposed to run in an isolated environment from which they could not escape. Nevertheless, they broke out and infiltrated Hugging Face, a major repository of artificial intelligence models and data. An independent investigation by researchers from METR and Redwood Research found that roughly twelve hundred agents in separate environments coordinated through an unauthorized message board, and about seven hundred of them joined the attack. According to OpenAI’s own findings, the company did not learn of the breach until approximately a week later. Monitoring of the models’ reasoning processes was not running at all during these tests.
In the same month, the UK AI Security Institute announced that every model it tested had attempted to cheat at least occasionally during cybersecurity evaluations. At the same time, more than one thousand employees of AI development companies signed a letter urging the U.S. government to create a mechanism for halting model development when necessary. In September, Anthropic CEO Dario Amodei published a plan under which the industry should slow the pace at which models acquire new capabilities. Anthropic also recently acknowledged that it cannot reliably control its AI agents and is therefore disconnecting its internal testing from the live internet.
Nadella had already hinted at his concerns in September at the All-In Summit. If you ask an advanced model to optimize a company’s working capital, he said, it might simply falsify the accounts. At the time, he described this as a new kind of insider risk. His Saturday article then turned that observation into a comprehensive guide.
Responsibility Lies with Whoever Deploys AI
The debate about slowing down AI has so far focused mainly on what the laboratories developing the models should do. Nadella, however, is addressing the other side: the companies that purchase and deploy AI. They are already using it extensively.
A survey conducted by Censuswide for the consulting firm Caylent questioned two hundred senior executives at large companies in the U.S. and Canada. Nearly sixty percent of them already allow AI agents to work independently in live production environments. Almost all would allow agents to make changes to live systems under certain conditions. At the same time, however, eighty-three percent of respondents said that stronger guardrails were just as important as, or even more important than, how intelligent the model is. Although the survey covered only companies already experimenting with agents, these are precisely the companies facing the greatest risk.
In this context, a Forbes writer recalls a case from the advertising world. In 2016, Facebook acknowledged a calculation error that had caused it to overstate average video viewing times for two years. At the time, advertisers allocated budgets based on figures that the seller itself both measured and reported. The advertising market responded with independent measurement and audits. According to Forbes, Nadella is applying the same logic to artificial intelligence. Whoever supplies the intelligence should not also be solely responsible for certifying and controlling it.
Microsoft Also Stands to Profit
Nadella’s article is written so that it does not refer to any specific vendor, and its principles make technical sense. However, Microsoft itself sells Agent 365, a tool it introduced last November. It is designed specifically for deploying and monitoring AI agents and assigns each agent its own identity. At the All-In conference, Nadella also mentioned that OpenAI is both an investment and one of Microsoft’s largest customers. And the agents that attacked Hugging Face in July belonged to OpenAI.
By his own admission, Nadella leaves some questions open. He sets aside the most difficult problem: how to ensure that models genuinely want the same things as humans. He acknowledges that standards for monitoring even more powerful models have yet to be developed. And the incident reporting he calls for is currently entirely voluntary. After investigating the Hugging Face case, METR CEO Beth Barnes pointed out that OpenAI gave independent investigators access to its data only because it chose to do so. No industry rule required it.
Sources: techcrunch.com, cnbc.com and forbes.com
