Solventum incident

https://cloudservices.3mhis.com degraded service affecting login

Major Resolved View vendor source →

Solventum experienced a major incident on September 22, 2023 affecting PRODUCTION (A) - 360 Encompass and PRODUCTION (A) - 360 Encompass and 1 more component, lasting 5h 1m. The incident has been resolved; the full update timeline is below.

Started
Sep 22, 2023, 12:08 PM UTC
Resolved
Sep 22, 2023, 05:10 PM UTC
Duration
5h 1m
Detected by Pingoru
Sep 22, 2023, 12:08 PM UTC

Affected components

PRODUCTION (A) - 360 EncompassPRODUCTION (A) - 360 EncompassPRODUCTION (A) - 360 EncompassNon-production (A) - 360 EncompassNon-production (A) - 360 EncompassNon-production (A) - 360 EncompassNon-production (A) - 360 EncompassPRODUCTION (A) - 360 Encompass

Update timeline

  1. investigating Sep 22, 2023, 12:08 PM UTC

    Starting at 07:38 Eastern, the https://cloudservices.3mhis.com service began to experience non-trivial latency. In the last few minutes, the service appears to have degraded further and some end-users may not be able to sign into 360 Encompass in the cloud. Affected users may see an alert that says, "This page isn't working right now. cloudservices.3mhis.com can't currently handle this request". The 3M cloud engineering team is investigating this issue.

  2. monitoring Sep 22, 2023, 12:44 PM UTC

    Cloud engineers have implemented a fix for this issue and MSO is validating it.

  3. monitoring Sep 22, 2023, 12:56 PM UTC

    Root cause analysis points to a DNS issue that was propagated to public name servers that may have cached bad/incorrect information. These caches will eventually clear, but apparent sporadic failures may continue to occur while they do.

  4. investigating Sep 22, 2023, 01:42 PM UTC

    DNS issues have been resolved, but some end-users continue to report Server 503 errors and this message "This page isn't working right now. cloudservices.3mhis.com can't currently handle this request". The 3M cloud operations team continues to investigate.

  5. investigating Sep 22, 2023, 02:11 PM UTC

    This issue continues to affect 3M Foundation Services and thus all 360 Encompass systems (hosted and on-premises) in various ways: end-users, primarily, will continue to see intermittent failures when logging into the application. The 3M cloud operations team continues to investigate this issue.

  6. monitoring Sep 22, 2023, 02:44 PM UTC

    The 3M cloud operations team has addressed several issues and believe that they will address the issue for all end-users. 3M MSO is monitoring this issue and see it being generally resolved for customer systems.

  7. monitoring Sep 22, 2023, 03:12 PM UTC

    The root cause of this issue appears to have been related to an acute web application firewall (WAF) failure that occurred around 0800 Eastern today, 2023-09-22. In response to the failure, the 3M cloud operations team failed over to a replacement resource, but DNS-caching caused some requests to continue to route to the failed resource. At approximately 1040 Eastern, the failed resource was recovered, restoring general functionality. Both the previously-failed (i.e. recently recovered) and the replacement resource will continue to operate throughout the day as cached DNS records clear. Please see Answer ID 1004195 on the 3M HIS support website for additional details about the effects and impacts of the outage: https://support.3mhis.com/app/answers/detail/a_id/1004195

  8. resolved Sep 22, 2023, 05:10 PM UTC

    This issue was fully resolved at 10:37 Eastern US time. Please see the previous update of this incident for root cause analysis. This issue was not related in any way to a 360 Encompass code update. Please also see Answer ID 1004195 for a summary of the effects of the outage: https://support.3mhis.com/app/answers/detail/a_id/1004195 This incident will no longer receive regular updates. Please contact your application administrator if you believe you may still be affected by this outage.