As a global cloud-using company's IT system is experiencing a blackout, it has been revealed that in order to resolve this, it is necessary to temporarily recover by restoring a backup or finding and deleting the 'C-00000291*.sys' file in safe mode.

On July 19, 2024 at 04:09 UTC, an incident occurred where Windows systems using CrowdStrike Falcon agents on cloud platforms became unresponsive or failed to start.

This has caused serious problems in various fields around the world, including aviation, shopping, delivery, and medical communities, with systems becoming unavailable.

To address this emergency situation, CrowdStrike has put together an emergency recovery plan.

According to CrowdStrike, the first thing to do is to understand the system status, but the symptoms are that Windows hosts that are not affected by the Falcon Sensor do not need to take action as the problematic channel file has been reverted.

Windows hosts that come online after 05:27 UTC are also not affected.

This issue does not affect Mac or Linux based hosts.

Channel files 'C-00000291*.sys' with timestamps greater than 05:27 UTC are reverted (good) versions.
<br /> Channel file 'C-00000291*.sys' with timestamp 04:09 UTC is the problematic version.

Please note that it is normal to have multiple 'C-00000291*.sys' files in your CrowdStrike directory, as long as one of the files in the folder has a timestamp greater than 05:27 UTC it is active content.

■ Solution on individual hosts

To troubleshoot a problematic system, boot Windows into Safe Mode or the Windows Recovery Environment.

In WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory on the OS volume.

Find and delete files matching 'C-00000291*.sys'.

Boot the host normally.

Note that Bitlocker encrypted hosts may require a recovery key.

■ Solution in public cloud or virtual environment

First, detach the operating system disk volume from the affected virtual server.

As a precaution against unintended changes, create a snapshot or backup of the disk volume before proceeding.

Attach and mount the volume to the new virtual server.

Go to the %WINDIR%\System32\drivers\CrowdStrike directory.

Find and delete files matching 'C-00000291*.sys'.

Detach the volume from the new virtual server.

Reattach the fixed volume to the affected virtual server.

A second method other than the above methodRolls back to a snapshot before 04:09 UTC.

More information can be found on the CrowdStrike homepage, MS Azure homepage, AWS homepage, etc.