Microsoft's investigation into last week's widespread Microsoft 365 outage has pinpointed the culprit: a defect introduced in a July update to its network gateway infrastructure. The company's preliminary post-incident report, shared with customers through the Microsoft 365 admin centre on 15 November and summarized by Charles Lamanna, Microsoft's executive vice president for Copilot, agents and platform, on X, reveals how a latent bug remained hidden for over three months before triggering a cascade of failures.

The root cause stemmed from lock contention within an internal Windows process. This defect caused servers to exhaust their CPU and memory resources under specific load conditions, rendering them unresponsive. Crucially, the July update had passed load testing without incident because the test environments used less capable hardware with fewer processor cores. The contention problem manifests more severely on modern machines with higher core counts, which explains why it escaped detection.

A perfect storm of three factors

The outage erupted when three conditions aligned simultaneously. First, Teams traffic failed over to the gateway infrastructure that also handles Exchange Online, dramatically increasing the load on those servers. Second, an unrelated issue from the previous day had already removed nearly 14% of the gateway's server capacity from operation. When the first server became unresponsive due to the lock contention bug, traffic shifted to the next available server, which then encountered the same defect. This pattern repeated across the infrastructure: within two hours, more than 80% of gateway servers had become unresponsive at the outage's nadir.

Recovery hampered by the bug's age

Restoring service proved unexpectedly difficult. As servers came back online, they immediately faced overwhelming traffic from queued requests, causing them to fail again. Microsoft's automated recovery systems were not architected to handle such widespread simultaneous failures. The company explained that because the defect originated from code changes made months earlier, standard rollback procedures and feature flags could not quickly reverse the problem.

Instead, engineers developed a new update that restored the Windows code path to its previous functioning state. They deployed this fix incrementally across server groups. Services including Teams, OneDrive, SharePoint, and Office applications recovered relatively quickly once they could route around the faulty gateways. Exchange Online faced different constraints due to scale considerations, which is why Outlook remained unavailable the longest.

According to the timeline published by The Register, the Teams failover began at 01:07 UTC on 12 November. The first customer support ticket arrived at 16:33 UTC that same day. Microsoft began applying its corrective update at 07:20 UTC on 13 November and issued its final customer communication at 04:15 UTC on 14 November. More than 1,000 Microsoft engineers and staff members participated in incident response calls coordinated by Lamanna and cloud chief Scott Guthrie, Bloomberg reported. On 13 November, Microsoft issued a public apology and clarified that neither cyberattacks nor AI agents were responsible.

Congressional scrutiny and global impact

https://x.com/Microsoft/status/2121282013600907325?ref_src=twsrc%5Etfw

The outage's reach extended far beyond typical business disruptions, affecting government agencies, judicial systems, hospitals, and enterprises worldwide. Downdetector recorded incident reports from 74 countries and territories, according to the Associated Press. The incident has drawn attention from U.S. lawmakers, with House Homeland Security Committee chair Andrew Garbarino telling Politico that he was requesting additional details from both Microsoft and CISA.

We need to understand what happened and how to make these systems more resilient

Andrew Garbarino

Microsoft intends to release its comprehensive post-incident report on 20 November, according to The Register.

Source: The Next Web