Microsoft Blames July Bug for Nearly Daylong Email Outage

Microsoft's preliminary review of the recent one day outage of Microsoft 365 says rerouted traffic spread failures across servers handling email connections.

TL;DR
  • Email Outage: Microsoft says a July software bug helped cause a nearly daylong outage of its hosted email service, Exchange Online.
  • Server Cascade: Redirecting users overwhelmed the gateway servers handling their connections, leaving more than 80% of that fleet unresponsive.
  • Uneven Recovery: Teams chats and meetings recovered sooner as traffic switched gateways; Exchange email remained tied to the failing infrastructure.
  • Repair Work: Microsoft restored earlier connection-handling code and added traffic controls.

Microsoft says a July software defect helped cause the November 12–13 outage that blocked users from its hosted Exchange Online mailboxes and delayed email. Its preliminary explanation shows how automatic traffic rerouting, normally used to keep services working, spread the failure across shared servers.

The defect affected gateway servers, which accept users’ connections to Microsoft 365 before passing them to services such as Exchange. A routine switch of traffic from Microsoft’s Teams chat and meeting service onto the gateway shared with Exchange, combined with reduced server capacity, triggered the fault months after its deployment.

Microsoft executive Charles Lamanna published an abridged account on November 16. Microsoft also shared a preliminary incident review.

How a Routine Traffic Switch Became a Cascade

Microsoft 365 uses two independent gateway systems so that services can switch between them during disruptions. At 01:07 UTC on November 12, an automated system moved Teams traffic from the alternative gateway to the primary one that also handled Exchange. The move was part of normal operations, about 15 hours before the main outage.

Nearly 14% of that gateway’s server capacity was also unavailable because of an unrelated problem from the previous day. Microsoft says the service had enough spare capacity to absorb that reduction. The changed traffic patterns and reduced capacity together exposed the July defect during peak US traffic.

Tasks inside a Windows process became stuck waiting on one another, a problem known as lock contention, until the servers exhausted processing power and memory. Requests piled up and machines stopped responding. The faulty code had been prepared specifically for Microsoft’s networking gateways.

Automatic traffic management then sent users to surviving servers. Those machines received more work, encountered the same defect and stopped responding in turn. In less than two hours, the failure had spread across the gateway fleet; more than 80% of its servers were unresponsive at the low point.

Internal barriers were supposed to isolate groups of servers and regions. A separate problem allowed the failure to cross those barriers and spread globally. Microsoft has not yet explained why those barriers failed.

 

Why the July Bug Escaped Testing

The gateway-specific change rolled out worldwide over several weeks from July and operated for more than three months without observed impact, according to Microsoft. Its load tests used a different traffic mix and emphasized weaker machines with fewer processor cores. The bug became worse on newer hardware with more cores, so those tests missed the more vulnerable configuration.

Why Email Took Longer to Recover

Teams, the OneDrive file-storage service, SharePoint collaboration services and Office apps could switch away from the faulty gateway, enabling earlier partial recovery. Microsoft says Teams chats and meetings worked for most of the outage. Exchange could not make the same switch, which the company attributes to the service’s scale.

Exchange Online hosts email and calendars; Outlook is one way to access those mailboxes. Microsoft says most users could not reach their Exchange mailboxes through Outlook or other clients, while many faced delayed delivery. Features in other Microsoft products that depend on Exchange were also affected, even where the rest of those products remained available.

Microsoft says the severity of the outage varied by region, connection method and time, and that access may have been partial or intermittent.

A January Microsoft 365 outage had also interrupted mail and collaboration services.

During the November outage, an unnamed Taiwanese financial company canceled a bond repurchase deal, a source familiar with the situation told the news agency Reuters. A second source said another firm used fax to confirm trades.

Restoring Servers Without Restarting the Failure

Returning servers to service initially brought the same problem back: waiting traffic overwhelmed them and they stopped responding again. Microsoft’s automated recovery was not designed for a failure this broad. Its usual rollback controls also could not quickly undo a change deployed months earlier.

Engineers instead developed an update restoring the previous connection-handling code. They took groups of servers offline, applied the update, restarted them and brought them back in batches. Deployment began at 07:20 UTC on November 13; after confirming stability, Microsoft expanded the repair, prioritizing regions within or entering business hours.

The preliminary review records an incident window from 16:20 UTC on November 12 to 16:10 UTC on November 13, or 23 hours and 50 minutes. It places global webmail’s return to expected availability at 14:17 UTC on November 13.

Microsoft’s corporate statement at 16:40 UTC that day still described restoring remaining customers and regions in stages. The company continued monitoring services and processing delayed mail.

Microsoft also extended the email-delivery window so queued messages could be delivered instead of being returned as undeliverable. The preliminary review marks that work complete, alongside additional capacity, monitoring of connection handling and traffic redirection, and controls on traffic distribution intended to prevent another cascade.

A comprehensive list of repairs is due with the final incident report on November 20.

Markus Kasanmascheff
Markus Kasanmascheff
Markus has been covering the tech industry for more than 15 years. He is holding a Master´s degree in International Economics and is the founder and managing editor of Winbuzzer.com.
Subscribe
Notify of
guest
0 Comments
Newest
Oldest Most Voted