Solventum incident
CrowdStrike disruption - major outage - all applications down/inaccessible/degraded
Solventum experienced a critical incident on July 19, 2024 affecting PRODUCTION (A) - 360 Encompass and PRODUCTION (A) - 360 Encompass and 1 more component, lasting 6d 17h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 19, 2024, 05:00 AM UTC
Beginning at 23:33 Central, outge alerts began to be received. We are investigating the issue.
- investigating Jul 19, 2024, 05:14 AM UTC
The AWS EC2 service appears to be impaired in the US-EAST-1 region resulting in general failures affecting all applications. We are continuing to investigate.
- investigating Jul 19, 2024, 05:37 AM UTC
Windows BSOD issues are occurring generally and affecting multiple services.
- investigating Jul 19, 2024, 05:57 AM UTC
An enterprise-wide issue affecting the Windows operating system has resulted in multiple key services failing. End-users are unable to access any managed applications at this time.
- identified Jul 19, 2024, 06:57 AM UTC
The root cause of the outage was related to a Crowdstrike update that caused outages worldwide. A fix has been applied to several affected systems and resolved their issues. Teams are working now to apply the fix generally. No ETA on general resolution.
- identified Jul 19, 2024, 09:31 AM UTC
Engineers have remediated the CrowdStrike issue and restored normal functionality on several environments, but due to the scope and scale of the issue, there is still no ETA on full recovery of all affected systems.
- identified Jul 19, 2024, 10:40 AM UTC
Engineering teams are still working on a general solution to recover platforms affected by the CrowdStrike issue. No ETA on full recovery.
- identified Jul 19, 2024, 12:31 PM UTC
Engineers have developed a general solution to resolve the issue across all affected systems. Teams are preparing to remediate individual systems presently and will begin the rollout at approximately 09:00 Eastern. No ETA on full recovery at this time.
- identified Jul 19, 2024, 04:08 PM UTC
Application administrators have begun the recovery of production systems and will be reaching out to emergency contacts to validate recovered environments. ETA target for full recovery of all affected environments is close of business today, Friday, 2024-07-19.
- identified Jul 19, 2024, 04:20 PM UTC
Production Coding in the cloud (CRS in the cloud, CITC) systems are expected to be restored within the hour.
- identified Jul 19, 2024, 05:54 PM UTC
Production coding in the cloud (CRS in the cloud, CITC) systems are fully recovered and released to end-users. 360 Encompass systems are being returned to end-users one at a time. Important! HL7 interfaces to managed 360 Encompass in the cloud systems will remain paused until systems are validated and released. Target time for full recovery of production systems is close of business, EDT.
- identified Jul 19, 2024, 07:38 PM UTC
Engineers continue to recover production 360 Encompass in the cloud systems. At present approximately 15% of production 360 Encompass in the cloud systems have been recovered, validated and released to end-users. All production coding in the cloud (CRS in the cloud, CITC) systems have been released (see previous update). ETA on full recovery for production systems is now close of business US Mountain time.
- identified Jul 19, 2024, 08:45 PM UTC
Approximately 30% of production 360 Encompass in the cloud systems have been remediated/validated.
- identified Jul 19, 2024, 09:59 PM UTC
Solventum cloud engineers have developed a proprietary rescue utility that will enable engineers to recover systems at a much higher rate of speed that previously. ETA on full recovery four to six hours from the time of this update.
- identified Jul 19, 2024, 10:31 PM UTC
Approximately 40% of production 360 Encompass in the cloud systems have been remediated/validated. ETA on full resolution for all systems is 3-5 hours from the time of this update.
- identified Jul 19, 2024, 11:39 PM UTC
Approximately 50% of production 360 Encompass in the cloud systems have been remediated/validated. ETA on full resolution for all systems is 2-4 hours from the time of this update.
- identified Jul 20, 2024, 12:40 AM UTC
Approximately 60% of production 360 Encompass in the cloud systems have been remediated/validated. ETA on full resolution for all systems is 2-3 hours from the time of this update.
- identified Jul 20, 2024, 01:48 AM UTC
Approximately 80% of production 360 Encompass in the cloud systems have been remediated/validated. ETA on full resolution for all systems is 1-2 hours from the time of this update.
- identified Jul 20, 2024, 02:58 AM UTC
Most production 360 Encompass in the cloud systems have been recovered and are available for normal use by end-users. A small number of systems that encountered more serious issues are still being repaired, validated, etc.
- monitoring Jul 20, 2024, 05:04 AM UTC
All production 360 Encompass in the cloud systems have been recovered and validated: end-users may access and use managed production environments as usual. All production Coding in the cloud (CRS in the cloud, CITC) have been recovered and validated. Non-production and legacy environments will be recovered during the coming days during North America US business hours. Important! Connexion Integrator Remote Agent installations on the local network may be affected by CrowdStrike while local IT teams resolve issues affecting your local network: this may cause delays in transmission of patient/visit data to 360 Encompass in the cloud. For down systems, please contact https://support.3mhis.com; for all other questions, please contact your assigned application administrator.
- monitoring Jul 22, 2024, 04:33 PM UTC
100% of production Coding in the cloud (CRS, CITC) environments have been operational since 15:38 EDT on Friday, 2024-07-19. 100% of production 360 Encompass in the cloud environments have been operational since 01:04 EDT on Saturday, 2024-07-20. Legacy, non-production and build environments are being recovered presently. Production deployments of legacy Codefinder/CRS environments are expected to be 100% recovered later today. The recovery of 100% of non-production 360 Encompass in the cloud environments is expected to be completed no later than 16:00 EDT on Wednesday, 2024-07-24. Build and "in progress" environments are expected to be 100% recovered by the end of the week.
- monitoring Jul 24, 2024, 10:59 PM UTC
Codefinder/CRS and Coding in the cloud (CITC): - 100% of production Coding in the cloud (CRS, CITC) environments have been operational since 15:38 EDT on Friday, 2024-07-19. - 100% of production Legacy Codefinder/CRS environments have been operational since 17:00 EDT on Tuesday, 2024-07-23. 360 Encompass in the cloud (360Eitc): - 100% of production 360 Encompass in the cloud environments have been operational since 01:04 EDT on Saturday, 2024-07-20. - 98.18% of non-production 360 Encompass in the cloud environments are operational as of 16:00 EDT on Wednesday, 2024-07-24. - Build and "in progress" environments are still expected to be 100% recovered by the end of the week. This includes deployments in Australia and Canada. Important! This incident will be marked "resolved" around COB EDT on Thursday, 2024-07-24. In the meantime: - Connexion Integrator Remote Agent installations on the local network may be affected by CrowdStrike while local IT teams resolve issues affecting your local network: this may cause delays in transmission of patient/visit data to 360 Encompass in the cloud. - For down systems, please contact https://support.3mhis.com. - For all other questions, please contact your assigned application administrator.
- resolved Jul 25, 2024, 10:05 PM UTC
This incident has been resolved.