Solventum incident
All regions - application monitoring service degraded
Solventum experienced a notice incident on August 3, 2023 affecting PRODUCTION (A) - 360 Encompass and PRODUCTION (A) - 360 Encompass and 1 more component, lasting 8h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 03, 2023, 12:09 PM UTC
Since approximately 01:00 Eastern time, application monitoring for 360 Encompass in the cloud and CRS in the cloud has been negatively affected and is not functioning correctly. This does not affect the end-user experience of any 3M Cloud Platform applications. Rather, that prevents the Managed Services Operations team from responding to systems that may be experiencing issues or outages. A cross-functional team is currently working to understand and resolve this issue.
- identified Aug 03, 2023, 12:41 PM UTC
Cloud engineers have identified an hourly database log backup that appears to be associated with the intermittent performance degradation affecting application monitoring. The team is currently developing a mitigation strategy.
- identified Aug 03, 2023, 02:53 PM UTC
A mitigation strategy for the degraded service has been applied and cloud engineers are observing application monitoring tools to determine whether they are able to function correctly again.
- monitoring Aug 03, 2023, 05:22 PM UTC
Root cause was determined to be an application making excessive calls to the 3M CoreSTS (https://cloudservices.3mhis.com) system and having the knock-on effect of increasing the amount of time the hourly database backup took to complete. The aggravating application has been addressed and cloud engineers are monitoring the system. 3M Cloud Platform application monitoring has resumed.
- resolved Aug 03, 2023, 08:10 PM UTC
This has been resolved and monitoring has been, at the time of the update, restored to full capacity for over three hours.