The first reports came from Australia early on a Friday: Windows computers stuck on the blue screen of death, unable to finish loading. By the time Europe and the United States woke up, the same thing was happening there.
The cause was a faulty software update from the security firm CrowdStrike. It was not a cyberattack, the company said. A defect in an update to its flagship product, Falcon Sensor, crashed any Windows machine running it. Microsoft said the problem began at 19:00 GMT on Thursday.
Airports, banks and hospitals
Airports took the worst of it. The Federal Aviation Administration ordered a ground stop early Friday, and by 18:00 GMT aviation data firm Cirium counted more than 4,000 cancelled flights, 3.9% of the total, though some may have been cancelled for other reasons. Some airports drafted in extra staff to check passengers in by hand.
Banks, payment systems and broadcasters were hit as well. Sky News and ABC Australia both went off air at points. JP Morgan Chase said it was working to restore its ATMs. Health services in Britain, Israel and Germany reported problems, and some operations were cancelled. In the US, the Social Security Administration closed its offices, and the Department of Education was among the affected agencies. President Biden was briefed, according to a pool report.
Microsoft separately reported an outage in an Azure cloud region covering much of the central US. A spokesperson told TechCrunch it had nothing to do with CrowdStrike.
The fix
CrowdStrike said a fix had been deployed. Rebooting could let a machine pick up the corrected file, but Microsoft said some users reported needing as many as 15 reboots. Where that failed, customers had to boot into Safe Mode or the Windows Recovery Environment and delete a file named “C-00000291*.sys” by hand, which meant physical access to each machine.
CEO George Kurtz apologised. “We’re deeply sorry for the impact that we’ve caused to customers, to travellers, to anyone affected by this, including our company,” he told NBC. He added that it “could be some time for some systems that just automatically won’t recover.”
CrowdStrike, founded in 2011, says it serves 29,000 corporate customers, including around half of the Fortune 500. Its shares fell about 12% on Friday.
The US Cybersecurity and Infrastructure Security Agency warned that it had seen threat actors using the outage for phishing and other malicious activity. Rachel Tobac of SocialProof Security urged people to confirm who they are talking to before acting. “Criminals will attempt to use this IT outage to pretend to be IT to you or you to IT to steal access, passwords, codes, etc.,” she said.
Sources
- What we know about CrowdStrike's update fail that's causing global outages and travel chaos | TechCrunch
- Microsoft IT outage: Global services slowly recovering after bug causes chaos
