Servers Down After CrowdStrike Update — How It Happened and How to Fix It
If you manage servers, you may have to cancel your weekend plans because a CrowdStrike update has caused the servers to BSOD/boot loop.
The incident does not appear to be a security incident or cyberattack and only affects Windows hosts. According to CrowdStrike, Linux and Mac were not affected.
The issue was first reported at 19:00 UTC on July 18 and was acknowledged by CrowdStrike in the early morning of July 19.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” said George Kurtz, CEO of CrowdStrike wrote on Twitter/X.
“This is not a security incident or cyber attack,” he added, “the issue has been identified, isolated and a fix has been implemented. We refer customers to the support portal for the latest updates and will continue to provide full and ongoing updates on our website.”
The good news is that a fix has already been found. The bad news is that servers that fail to boot will likely require manual intervention. CrowdStrike provided the following instructions on how to resolve the issue.
- Boot Windows into Safe Mode or Windows Recovery Environment
- Navigate to the folder C:\Windows\System32\drivers\CrowdStrike
- Find the file corresponding to C-00000291*.sys* and delete it
- Boot the host normally
Microsoft later further advice:
- We recommend that customers who are able to do so restore from a backup prior to 19:00 UTC on July 18
- You can also try to repair the OS disk offline.
- Connect a disk to the virtual machine for offline repair (encrypted disks may require further instructions)
- Once the drive is connected, delete the file Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys
- We can confirm that the affected update has been pulled by CrowdStrike. Customers who are still experiencing issues should contact CrowdStrike for additional assistance.
Who is affected by the CloudStrike update?
The CrowdStrike update affected Windows devices and virtual machines running Windows Client and Windows servers running the CrowdStrike Falcon agent. Personal PCs running Windows are not affected.
It’s not yet known exactly how many machines have been affected, but it’s already had a major impact around the world, particularly in Europe, where Visa, Amazon and Microsoft are all reporting issues. There have also been reports of airlines and hospitals having issues. The extent of the impact won’t be known until later today.
How do I resolve the CrowdStrike issue?
Basically you should delete the file corresponding to C-00000291*.sys
You can do this by
1. Boot Windows into Safe Mode or Windows Recovery Environment
2. Navigate to the folder C:\Windows\System32\drivers\CrowdStrike
3. Find the file corresponding to C-00000291*.sys and delete it
or
You may need to manually remove/update the OS disk
What is CrowdStrike?
CrowdStrike is a cybersecurity company behind software used by some of the world’s largest companies and institutions, including hospitals, airports, banks, and many Fortune 500 companies.
