“Unable to restore access to the resources and data.” That line, from an Amazon Web Services health dashboard update posted on September 15, is how the company told customers that some of what they stored in the Middle East is gone for good. Reuters reported it first. The cause is six months old: Iranian drone strikes in early March that damaged AWS facilities in Bahrain and the United Arab Emirates during the conflict that followed U.S.-Israeli attacks on Iran.
Bahrain and the UAE
The damage was uneven. In the UAE region, customer data was irretrievably lost in one of three availability zones, mec1-az2. Anything hosted exclusively there cannot be recovered. AWS says it is still working on regional resources and on the other two zones, mec1-az1 and mec1-az3, and has promised customers updates on restoring service in the coming months.
Bahrain fared worse. AWS said it cannot restore access to resources and data hosted exclusively in that region, across all three of its availability zones. “The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand,” the update said.
AWS has not said how much data was lost or how many customers were hit.
What redundancy could not do
Each availability zone is serviced by one or more data centers and is built as an isolated location inside a region, so customers can spread an application across several. That protects against a fire or a power failure in one building. It does not protect against a war that disables several zones at once. CircleID’s analysis made the point plainly: customers who need to survive that class of failure need recoverable copies or workloads outside the affected region.
Many did exactly that. AWS had advised affected customers to move workloads to other regions and restore from remote backups, and says most re-established operations elsewhere by restoring backups or copying data that remained accessible.
A long outage gets longer
By April 30, AWS was warning that restoration would take several months, and it had suspended billing in the affected regions. Recovery in the UAE continues, with replacement of affected infrastructure expected to take additional months.
Sources
- Iran strikes on Amazon data centers caused permanent loss of customer data
- AWS Says Iran Strikes Caused Permanent Customer Data Loss in Bahrain and UAE
