A Friday that began with blue screens
Before most of the United States had woken up on July 19, 2024, American Airlines, Delta and United had stopped their flights. The FAA told airlines that “all… flights regardless of destination” were grounded because of communication problems. By 5:00 a.m. Eastern, American said it had safely re-established its operation. Elsewhere the day was only getting started.
The cause was a software update from the security company CrowdStrike, aimed at its Falcon Sensor product on Windows machines. Microsoft later said that a little under 1%, or about 8.5 million, Windows devices were hit. Each one crashed into a blue screen and then fell into a reboot loop.
Not an attack
CrowdStrike chief executive George Kurtz moved quickly to say what it was not. “This is not a security incident or cyberattack,” he wrote on X, describing a defect in a single content update for Windows hosts. Britain and France reached the same early view. France’s national cybersecurity agency, ANSSI, said there was no evidence of a cyberattack.
Microsoft pointed to a problem with Windows virtual machines running the CrowdStrike agent, which could hit a bug check and stay stuck in a restart state.
What it knocked over
The list ran across borders. In Germany the supermarket chain Tegut shut more than 300 stores because the checkouts did not work. Eurowings suspended its domestic German flights and flights to and from the UK until at least 3 p.m. GMT, with more than 50 flights expected to be delayed or cancelled. Spain’s airport operator Aena said all its airports were open but some processes ran with delays, mainly at check-in. Mercedes-Benz said its global production network was partly affected. The Paris Olympics organising committee activated contingency plans a week before the Games opened.
Markets reacted too. CrowdStrike shares were down 20% in pre-market trading on Wall Street, while Microsoft slipped 1.4% early in the day, according to DW’s live coverage.
How the bad file got through
CrowdStrike’s early findings, posted to its guidance hub the following Wednesday, put the blame on a bug in the content validator of the Falcon platform. The validator missed an errant configuration update for the Windows sensors. And because the update was classed as Rapid Response Content instead of Sensor Content, it did not go through the same pre-release testing that Sensor Content would have faced.
A spokesperson told TechTarget that the faulty channel file template was removed from Channel File 291 at 0527 UTC on July 19. The logic error in the validation system had been fixed and was being tested, with release to production planned for early the following week.
A slow cleanup
Fixing the machines was the hard part. Each affected device needed a manual restart in safe mode so the defective file could be deleted. Microsoft and CrowdStrike published workarounds and recovery tools, and disruption carried on into the next week.
By the evening of July 25, Kurtz said more than 97% of Windows sensors were back online. He apologised directly: “I am deeply sorry for the disruption this outage has caused and personally apologize to everyone impacted.” CrowdStrike, he added, would not rest until every affected customer was restored.
Sources
- CrowdStrike: 97% of Windows sensors back online after outage | TechTarget
- Global cyber outage blamed on security software update
