Crowdstrike Update Causes Major Global Windows Outage

Thousands of Windows systems worldwide have been impacted by a crash and subsequent BSOD (Blue Screen of Death).

Machines are crashing following a Crowdstrike antivirus definitions update (a channel file specifically). This update has supposedly been automatically pushed to Windows devices running the Crowdstrike Falcon Sensor product.

Who's Impacted

This impacts Windows systems only, OSX and Linux hosts are not affected.
Hospitals, airlines, TV broadcasters, banks and retailers are among the many impacted by this. Anyone running Crowdstrike Falcon on Windows could potentially see the BSOD.
American Airlines stated the following on their site. "A third-party software outage impacted technology systems worldwide, including at American. Our flights have resumed and we’re working diligently to minimize disruptions. We’ve issued a travel waiver so you can change your trip online or in the American app." Chase Bank and Bank of America have had outages, preventing login and thereby trading. Possibly related, health provider Cleveland Clinic's site appears to be down, showing just a cached snapshot of the site as provided by Cloudflare CDN.

Websites aren't the only entities affected. Gas stations, restaurants and even 911 call centers have been taken down by this issue in several US states.

Not a Cyberattack!

Though the widespread impact certainly makes it feel like an attack, as stated this issue is caused by a faulty update. Crowdstrike is working to restore customer machines impacted by this and has issued a blog post detailing recovery steps.

While they are pushing out a reverted channel file, some systems may require manual intervention. This will no doubt delay recovery for certain businesses and public services.
The manual steps from their post are as follows:

Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
3. Locate the file matching C-00000291*.sys, and delete it.
4. Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.

Aftermath and Future Prevention

You can be certain there are and will be many questioning the processes involved in Crowdstrike's QA and code quality review. There should also be scrutiny around the AV definition updates process.

The challenge is, modern IT systems depend on these updates to ensure they are protected from the latest emerging threats. This means automated pushes of such changes are expected and required to maintain a security posture. With that in mind, the current update process is not likely to go away. While a manual update process would probably have reduced impact of this outage, that has the potential to open systems to exploits as companies either fail to keep up or simply ignore the updates.

Focus should be placed on improving CrowdStrike's software quality testing. Installation of this channel file should have been done on test systems where the issue could've been detected early.
Judging by today's events, it appears as though very little pre-release testing was actually done. A troubling consideration.

I'll be watching this issue closely, especially the follow-up and response.