Solventum experienced a minor incident on December 14, 2021 affecting PRODUCTION (A) - 360 Encompass and PRODUCTION (A) - 360 Encompass and 1 more component, lasting 20d 20h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 14, 2021, 06:50 PM UTC
3M HIS published a KPI detailing the vulnerability and 3M HIS's response here: https://support.3mhis.com/app/answers/detail/a_id/1002040 Related KPI https://support.3mhis.com/app/answers/detail/a_id/1002041 will be updated shortly to include specific patch files and patching instructions for 360 Encompass. The 360 Managed Services Operations team intends to apply the patch to non-production and non-live production environments as soon as it is available from 3M HIS development and will plan to apply the patch to production environments during Wednesday (12/15) night's scheduled maintenance window. Please subscribe to this update for additional information as it becomes available.
- identified Dec 15, 2021, 02:52 PM UTC
Internal 3M HIS testing has been completed and KPI https://support.3mhis.com/app/answers/detail/a_id/1002041 has been updated with specific steps for patching 360 Encompass environments affected by this issue. The Managed Services Operations team will be updating non-production environments this morning and throughout the day and users may notice that non-production systems are temporarily inaccessible. Please contact your 360E system administrator if you have any concerns about access to non-production environments this morning. Production environments will be patched during this evening's scheduled maintenance window.
- identified Dec 16, 2021, 12:41 AM UTC
The Managed Services Operations team is currently patching production 360E environments using KPI https://support.3mhis.com/app/answers/detail/a_id/1002041 This incident will be updated at the end of tonight's scheduled production maintenance window to confirm that all production environments have been patched.
- monitoring Dec 16, 2021, 02:26 AM UTC
All production 360 Encompass environments have been patched to include hotfix 1002041: https://support.3mhis.com/app/answers/detail/a_id/1002041 No further updates on this incident are expected at this point, but it will remain open and in "Monitoring" status at this URL until Friday (12/17) afternoon for administrative purposes. For specific questions about your managed 360 Encompass or CRS system, please contact your 3M HIS system administrator directly.
- monitoring Dec 16, 2021, 04:05 PM UTC
All managed instances (including inactive, dev-only and non-user-facing instances) of 360 Encompass were patched during this morning's scheduled non-production maintenance window. As mentioned previously, for administrative purposes, this incident will remain open and in the "Monitoring" status until Friday (12/17) afternoon. Please contact your 3M HIS 360 Encompass System Administrator directly if you have any specific questions about managed systems.
- monitoring Dec 20, 2021, 03:08 PM UTC
On Friday evening (12/17), 3M HIS development updated hotfix 1002053 to include additional critical patching instructions to bring log4j up to version 2.16 on 360 Encompass application servers: https://support.3mhis.com/app/answers/detail/a_id/1002053 The Managed Services Operations team is making preparations to apply this update to production environments during this week's scheduled maintenance window.
- monitoring Dec 20, 2021, 04:06 PM UTC
Non-prod/non-live patching update.
- monitoring Dec 20, 2021, 04:08 PM UTC
Starting at noon Eastern, today 12/20, the Managed Services Operations team will be patching non-production and non-live 360 Encompass application instances to version 2.16 of log4j in accordance with KPI https://support.3mhis.com/app/answers/detail/a_id/1002053 Systems are expected to be unavailable for approximately 15-30 minutes during patching. Please contact your 360E system administrator if you have any specific questions or if you are unable to access your non-production/non-live 360E environment for more than 30 minutes today.
- monitoring Dec 21, 2021, 06:37 PM UTC
Non-production and non-live production 360 Encompass application servers have been patched from 2.15 to 2.16. As planned, production servers will be patched from 2.15 to 2.16 during the scheduled maintenance window on Wednesday night, 12/22. For administrative purposes, this incident will remain open until at least 2021-12-29. Please contact your 3M HIS 360 Encompass system administrator with any specific questions about 3M HIS managed resources.
- monitoring Dec 23, 2021, 02:23 AM UTC
As of 21:22 Eastern time, all production 360 Encompass application servers have been patched to version 2.16 of log4j. For administrative purposes this incident will remain open until close of business, Mountain time, Thursday, 2021-12-23. For specific questions about your managed 360 Encompass or CRS system, please contact your 3M HIS system administrator directly.
- monitoring Dec 28, 2021, 06:31 PM UTC
On Monday 12/27 3M HIS development released a KPI that patches 360E environments with log4j 2.16 to 2.17: https://support.3mhis.com/app/answers/detail/a_id/1002064 Application of the KPI requires the January General release of 360 Encompass (2.11.20.0). Non-production/non-live managed instances will be patched this week (12/30) after they are upgraded to 2.11.20.0; production instances will be upgraded to 2.11.20.0 and patched during the scheduled maintenance window on Wednesday, 2022-01-05.
- monitoring Dec 31, 2021, 03:24 PM UTC
Following scheduled non-production maintenance on Thursday, 12/30, all non-production 360E environments have been upgraded to the January General (21.11.20.0) regulatory release of 360 Encompass and patched to version 2.17 of log4j according to KPI 1002064 (https://support.3mhis.com/app/answers/detail/a_id/1002064). Production systems will be upgraded to January General and patched to version 2.17 of log4j during scheduled maintenance this Wednesday, 2022-01-05. Please contact your 3M HIS 360 Encompass System Administrator directly if you have any specific questions about managed systems.
- resolved Jan 04, 2022, 03:00 PM UTC
Following last week's work to patch systems up to version 2.17 of log4j, this issue is resolved.