Cirium counted 5,078 flights cancelled worldwide on Friday, 19 July 2024. That was 4.6% of everything scheduled to fly, and the cause was not weather or a strike. It was a single file pushed to Windows machines by the Texas-based security company CrowdStrike.
The first reports came in Friday morning. Airlines, media companies and government agencies watched their Windows systems drop into the “blue screen of death”. Many people feared a cyberattack. Security experts quickly pinned it on a botched update to CrowdStrike’s Falcon platform instead.
What CrowdStrike said
CEO George Kurtz went to X to say the company was working with customers hit by “a defect found in a single content update for Windows hosts”. Mac and Linux hosts were fine. “This is not a security incident or cyberattack,” he wrote. Later that morning he apologised: “We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption.”
A company blog post filled in the mechanics. CrowdStrike had released a sensor configuration update just after midnight EST. These “channel files” change several times a day, in response to new attacker techniques. This one was meant to counter newly observed malicious activity, but it carried a logic error that crashed the systems it reached. Brody Nisbet of CrowdStrike Overwatch put it more bluntly on X, calling it a “faulty channel file, so not quite an update.”
A repair that has to be done by hand
Stopping the bad file from spreading was the easy part. Fixing machines already stuck in a boot loop was not. The workaround meant rebooting each Windows system in safe mode, deleting the defective file, and restarting.
John Hammond of Huntress told TechTarget it was “a very manual process” that had to be done “at the physical location of the computer, by hand, for every computer impacted”. Gabe Knuth of Enterprise Strategy Group added more snags. Local admin rights are needed, and where disks use BitLocker encryption the recovery key must be on hand. Some customers found the server holding those keys had crashed too.
Microsoft published guidance for restoring Windows 365 cloud systems, and its Azure status page said some customers recovered after several restarts, with as many as 15 reported. AWS posted its own recovery steps.
The wider damage
The Guardian called it the “largest outage in history”. American Airlines, Delta and United were among the carriers affected in the US, and Berlin airport briefly halted all flights. In the UK, Sky News went off air for a time, GP practices lost access to patient records, and the Royal Surrey NHS Trust declared a critical incident and cancelled radiotherapy appointments. Israel’s health ministry said 16 hospitals were hit. Portland’s mayor, Ted Wheeler, issued an emergency declaration. CrowdStrike’s shares fell by as much as 13% during the day.
Security researcher Troy Hunt said it would be the largest IT outage in history, and compared the scare to the Y2K bug, “except it’s actually happened this time.”
The kernel question ran through the expert reaction. Security tools like Falcon sit very deep in Windows, so a corrupt driver can do serious harm. Knuth said such incidents are “always a possibility”. Maxine Holt of Omdia was harsher on the business side: “This is very bad for CrowdStrike from a business perspective.”
By September 2024, CrowdStrike’s Adam Meyers told a House subcommittee that more than 8.5 million Windows devices had been disabled. The company said it would now treat content updates as code, and let customers choose when to receive them.
Sources
- Defective CrowdStrike update triggers mass IT outage | TechTarget
- CrowdStrike tells Congress of two process changes to address July outage incident | The Record from Recorded Future News
- Slow recovery from IT outage begins as experts warn of future risks | The Guardian
