Intermittent server 500 errors affecting 360 Encompass

Update timeline

1. investigating Nov 04, 2024, 03:23 PM UTC

   Support and operations teams are receiving an unusually high number of end-user reports of 500 Server Error issues. Root cause is unclear, but multiple EHR's and integrated workflows (i.e. "launches") appear to be affected.

2. investigating Nov 04, 2024, 04:58 PM UTC

   Application and production support teams continue to investigate intermittent 500 Server Error issues in production systems. No root cause has been determined at this point, but the affects appear to apply to 360 Encompass generally: no specific EHR/EMR, deployment/hosting, or integrated session ("launch") type appears to be affected more than any other.

3. investigating Nov 04, 2024, 05:52 PM UTC

   Application and production support teams continue to investigate an issue that appears to affect all 360 Encompass systems (including some 360 Encompass in the cloud systems). Affects are intermittent, but end-users report sporadic 500 Server Errors during (primarily) outpatient coding workflow.

4. investigating Nov 04, 2024, 06:50 PM UTC

   Several 360 Encompass and CRS/Codefinder systems (including some managed 360 Encompass in the cloud and Coding in the Cloud systems) continue to be affected by intermittent issues where end-user coders receive a "500 - Internal Server Error" response from the encoder during the integrated coding session. Application and production support teams are continuing to develop a root cause.

5. investigating Nov 04, 2024, 09:17 PM UTC

   Continued reports of intermittent 500 Internal Server Errors and acute slowness following button-clicks in coder workflows. Some affected systems have resolved issues by clearing end-users' Chromium Embedded Framework or Citrix browser caches, suggestive of a potential relationship to the North America US time change. No consistent root cause identified at this point.

6. investigating Nov 04, 2024, 11:27 PM UTC

   No root cause has been identified for the issue, though reports have become increasingly sporadic/infrequent. In some cases, the issue appears to have resolved itself. In others, clearing end-user browser cache (see previous update) has resolved the issue. In some cases, changes to idle timeout settings have been recommended as a workaround. This has been an issue with a large scope, but a very limited severity/impact. Accordingly, this incident will remain open but will only be updated during North America US business hours.

7. identified Nov 05, 2024, 01:41 PM UTC

   Application support teams have identified an apparent link between affected end-users and browser sessions that were either active during or otherwise persisted through the 2024-11-03 time change. At this point affected end-users are advised to clear the caches of any browsers that have not been cleared/initialized since the time change on Sunday.

8. resolved Nov 06, 2024, 04:48 PM UTC

   Root cause of intermittent slowness and 500 issues following the time change appears to have been due to information cached by browsers that were active during (or persisted through) the 2024-11-03 time change.