Azure Went Down and People Actually Noticed

A Substantial Part of Microsoft's Azure Cloud Platform Went Down Earlier this Week, Leaving Customers with Deployments in the US, Europe and Asia, Unable to Use Azure for about 11 Hours. Microsoft has Explained the Problem. An Update was Made to Azure Storage that Caused the Storage Front-End Servers to Get Stuck in an Infinite Loop, Leaving them Unable to Service any Requests. Everything Else that Depended on Storage subsequently Keeled Over. Although the Bad Update was Reverted, it Took a while for the Subsequent Reboots to Ripple through the System and Get Things Working Again. The Update had been Tested before the Broad Rollout but, the Testing had Only been Performed on Azure's Table-Based Storage. The Infinite Loop Error was Discovered in Azure's Blob-Based Storage. This was Compounded by a Rollout that was More Rapid than Normal; Updates such as this, Should be Deployed in Limited Batches to Avoid Taking Out Multiple Azure Data Centers Simultaneously.


In the Future, Microsoft Says it's Going to Make its Deployment Tools Enforce Slower Batched Rollouts and the Company is also Going to Work to Improve Recovery Times. But, there is perhaps a Small Silver Lining in this Particular Cloud. Amazon's AWS has Suffered Widespread Outages in the Past and Twitter has Immediately Lit Up with People Complaining about it. Azure Outages in the Past Haven't had Anything like the Same Visibility. This Week's Azure Outage, however, Provoked a Large and Angry Response — with Many Customers Upset, Not just at the Outage itself but, at the Azure Service Status Dashboard, which was apparently Being Deeply Misleading during the Downtime. While Complaints about Downtime perhaps Aren't the Best Advertisement a Service Provider Could Hope for, it's Much Better than Not Having Complaints because, Nobody's Using your Service in the First Place.




Info Source:

http://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption