Cyderes experienced a major incident on July 19, 2024 affecting Third-Party Data Source, lasting 3d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 19, 2024, 07:42 AM UTC
Cyderes has been made aware of an issue with a Crowdstrike Content update deployment which may cause a Blue Screen of Death (BSoD) for servers across the globe running Falcon Sensor. This issue is not directly impacting Cyderes’ platform nor our ability to continue providing Managed Services to your firm. This issue does not appear to be impacting MacOS or Linux-based hosts. Endpoints that did not receive the faulty Content update were not impacted. Crowdstrike has reverted their Content update but if you are experiencing crashes and unable to stay online, Crowdstrike's suggested workaround(s) to correct the issue include the following: Workaround Steps for individual hosts: Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: • Boot Windows into Safe Mode or the Windows Recovery Environment o Note: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation. • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory • Locate the file matching “C-00000291*.sys”, and delete it • Boot the host normally Note: Bitlocker-encrypted hosts may require a recovery key. In the spirit of community sharing, some of our clients have had success with hosts running Bitlocker by performing the following (Sharing for awareness): manage-bde -unlock X: -Password or manage-bde -unlock X: -RecoveryPassword. ***Remember to replace the letter “X” with the drive letter of the BitLocker encrypted drive Workaround Steps for public cloud or similar environment including virtual: Option 1: • Detach the operating system disk volume from the impacted virtual server • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes • Attach/mount the volume to to a new virtual server • Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory • Locate the file matching “C-00000291*.sys”, and delete it. • Detach the volume from the new virtual serverReattach the fixed volume to the impacted virtual server Option 2: • Roll back to a snapshot before 0409 UTC. If this issue affected your Single Sign-On Server, please follow the above steps from Crowdstrike first in order to mitigate the issue, as Cyderes is unable to create local accounts on machines suffering from this issue. For further information, or for the absolute latest guidance on how to remediate, please contact Crowdstrike Support.
- resolved Jul 22, 2024, 01:57 PM UTC
This incident has been resolved.