Microsoft Blames 16-Hour Outlook Outage on Overheating Data Center

The 16-hour Outlook outage on March 12 has been attributed to a firmware upgrade that went wrong. Microsoft (News - Alert) is in the middle of transitioning its Hotmail user base over to its new Outlook service.

Such an outage is considered rare for this type of high caliber Web service that required human intervention to get Outlook back online making for a lengthy restoration process, company officials said.

Detailing the outage in a post-mortem blog post, Microsoft officials said there was a service interruption that affected some users’ access to a small part of the SkyDrive service, but primarily Hotmail.com and Outlook.com.

While performing its regular process of updating the firmware on a core part of Microsoft’s physical plant, a failure occurred that resulted in a “rapid and substantial temperature spike” in the data center, according to Arthur de Haan, vice president of test and service engineering for Microsoft’s Windows division.

“This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter,” de Haan explained in the post. “These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access.”

This particular area of the data center he referred to houses parts of the Hotmail.com, Outlook.com and SkyDrive infrastructure.

Microsoft brought back access in waves throughout the evening of March 12 with the majority of the affected mailboxes fully restored before midnight and the rest completed by 5:30 a.m. on March 13.

“Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration,” de Haan said.

Could this kind of outage happen again in the future? de Haan said post-incident that Microsoft is working to make sure it doesn’t.

“Now that we’re through the resolution, we’re also hard at work on ensuring this doesn’t happen again,” he said.