# Microsoft outage: A cascading failure that started with an undetected bug

DevFeed: [Microsoft outage: A cascading failure that started with an undetected bug](<https://devfeed.tech/articles/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-136600.md>)

Original publisher: [Read original article](<https://www.networkworld.com/article/4249392/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-2.html>)

Author: Evan Schuman

Published: 2026-11-17T00:02:57Z

Content type: news

Language: en

Sources: [Network World](<https://devfeed.tech/sources/network-world.md>)

Topics: [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [microsoft 365](<https://devfeed.tech/topics/microsoft-365.md>), [test](<https://devfeed.tech/topics/test.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Microsoft](<https://devfeed.tech/topics/microsoft.md>), [Microsoft Teams](<https://devfeed.tech/topics/microsoft-teams.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [bug](<https://devfeed.tech/tags/bug.md>), [defect](<https://devfeed.tech/tags/defect.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [exchange-online](<https://devfeed.tech/tags/exchange-online.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [global](<https://devfeed.tech/tags/global.md>), [global-outage](<https://devfeed.tech/tags/global-outage.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-report](<https://devfeed.tech/tags/incident-report.md>), [independent](<https://devfeed.tech/tags/independent.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [issue](<https://devfeed.tech/tags/issue.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-outage](<https://devfeed.tech/tags/microsoft-outage.md>), [networking](<https://devfeed.tech/tags/networking.md>), [outage](<https://devfeed.tech/tags/outage.md>), [storytype-news](<https://devfeed.tech/tags/storytype-news.md>)

## AI overview

A Microsoft 365 gateway software defect, deployed globally for months without impact, surfaced when Teams failed over to the primary gateway, increasing and changing traffic while nearly 14% of gateway capacity was already unavailable. Microsoft's load tests missed the defect because they used different traffic and SKU mixes and tended to use weaker, lower-core hardware; the report says more cores and newer hardware worsened contention. Requests accumulated, and automatic rerouting spread load across fewer healthy servers, causing a fleet-wide cascade that left more than 80% unresponsive at the low point. Recovery automation and normal rollback, flighting and feature flags could not quickly restore service. The preliminary report did not explain the unrelated capacity issue, and the article notes that Microsoft had yet to provide customers concrete reassurance against recurrence.

## Source excerpt

Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on. The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. "A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," the report said, adding, "this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers." Its deployment began in July, and, the report said, "for more than three months, the change was deployed worldwide without impact." Microsoft admitted that, although it did test the change before deployment, its test environment didn't match reality. "Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs," it said. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores." A surge in traffic caused by unrelated issues also helped trigger the problem and delay it from being fixed, the company explained. "Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability," Microsoft said. "This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns." To make things worse, the report said, "nearly 14% of server capacit