Speaking at The New York Times' DealBook Summit in New York on Wednesday, Satya Nadella outlined a troubling dynamic: machine learning systems are uncovering flaws in software at a pace that outstrips the ability of organizations to test and deploy fixes responsibly. He anticipates this acceleration will produce more widespread outages similar to the recent Outlook incident.

Nadella acknowledged Microsoft's own outage but clarified it stemmed from a different cause. "This was a latent bug that was there. It got invoked and, you know, it's one of those perfect storms," he explained. Later, speaking with Jim Cramer on CNBC's Mad Money, he conceded the company should have taken additional precautions: "Of course, we should have done the testing before. We should have figured out how to have the bulkheads such that you didn't have a global outage."

Patches with no time to test

The real danger, according to Nadella, emerges from AI tools that identify security vulnerabilities. Historically, critical patches for banking systems underwent a week of rigorous testing before deployment. Today's threat landscape has inverted this timeline: attackers exploit flaws the moment they become public, forcing organizations to push fixes directly into live systems without adequate validation.

On the very day of his remarks, Nadella noted that three critical patches had been released for open-source software. "The patch needs to go instantly from being available to being patched in production, because it's being exploited in the wild. So that's going to create its instability," he said.

Recent major outages in the United States have been attributed by banks and the NSA to hastily applied patches addressing AI-discovered flaws. Nadella also pointed to a broader erosion of security posture: the baseline for cyberattacks has fundamentally shifted downward. With AI models now accessible to many actors, anyone can effectively operate as a persistent, nation-state-caliber threat.

When Sorkin asked whether this represented a temporary condition, Nadella drew a parallel to the Y2K bug but acknowledged uncertainty given the rapid advancement of model capabilities. He highlighted an additional vulnerability: much of the world's critical infrastructure runs on firmware that cannot be patched. This includes chillers, heating and cooling systems, and transportation networks connected to the internet.

'A world of hurt'

Nadella shifted focus to the alignment problem in AI development. He noted that in a previous conversation with 20VC, he had argued that alignment would not emerge from scale alone. Sorkin mentioned that Sam Altman, who had spoken earlier at the summit, held a different view on this premise.

Nadella acknowledged learning about scaling laws from OpenAI's founders and conceded that scaling had delivered results. However, he stressed that alignment and practical debugging methods for models have not kept pace. "I mean, if it doesn't happen, we are in a world of hurt," he said, expressing hope that Altman and Greg Brockman would prove correct.

His proposed solution centers on what he terms a trust architecture. Models that reason using neuralese—an opaque internal language humans cannot interpret—should be prohibited, he argued. Every model's reasoning process must be accompanied by tamper-proof audit logs that other systems can examine, since models themselves may attempt to conceal their operations. He commended OpenAI for its practice of monitoring models throughout training.

Under current frameworks, the entity deploying an AI system bears responsibility for its actions. Nadella expects insurance companies to quantify this liability and demand access to the logs as a condition of coverage.

Cheap tokens, borrowed frontier

Sorkin raised a question about the long-term viability of OpenAI and Anthropic, both of which have received Microsoft investment, if less expensive models can perform equivalent tasks. Nadella responded that routine office applications no longer require frontier-grade models. Copilot already routes requests among OpenAI, Anthropic, and Microsoft's own MAI models based on the task at hand.

He emphasized that his primary stake remains in Microsoft itself. "Arguing for some massive pricing power of tokens is betting against somehow US GDP," he said.

Microsoft's arrangement with OpenAI provides royalty-free access to a frontier model through 2032. This arrangement permits Microsoft to develop MAI models that trail the frontier by three to six months while consuming only 5% to 20% of the computational resources. Nadella characterized Anthropic and OpenAI as playing "a vertical stack game", while Microsoft pursues "a model system game".

Agents, jobs, and Washington

Nadella predicted that AI agents will fundamentally reorganize economic flows on the internet, comparable to the shift from Yahoo to search-based discovery. Websites dependent on advertising revenue from human traffic will need to adapt. When asked about companies that restrict agent access, such as Amazon, Nadella suggested they would cede market share to competitors. "If you don't accept the agent, somebody else will. If Amazon doesn't, Shopify will," he said.

Microsoft's recent voluntary separation offers extended to 60,000 employees were framed as empowering individuals with agency. While software engineering remains vibrant, the nature of the work has transformed. The team developing Microsoft's Cowork agent consists of 11 engineers who function as "a software factory that builds the product". This transition will unfold more gradually than many anticipate, Nadella cautioned.

Sorkin observed that Nadella has maintained distance from the Trump administration. Nadella explained that his focus is on earning Microsoft's license to operate through the quality of its offerings. "I just don't want to be in Washington. I just want to be in Redmond, Washington," he said.

Source: The Next Web