Healthise incident

Performance Issues

Major Resolved View vendor source →

Healthise experienced a major incident on April 1, 2021, lasting 16h 51m. The incident has been resolved; the full update timeline is below.

Started
Apr 01, 2021, 10:03 PM UTC
Resolved
Apr 02, 2021, 02:55 PM UTC
Duration
16h 51m
Detected by Pingoru
Apr 01, 2021, 10:03 PM UTC

Update timeline

  1. investigating Apr 01, 2021, 10:03 PM UTC

    We are currently investigating this issue.

  2. identified Apr 01, 2021, 10:23 PM UTC

    We've identified the issue as being caused by a DNS outage of our vendor (Microsoft Azure) affecting a subset of customers. They are actively working to restore service and we will provide regular status updates as we receive them. https://status.azure.com/en-us/status

  3. monitoring Apr 01, 2021, 10:43 PM UTC

    Application performance has stabilized and we will continue to monitor.

  4. resolved Apr 02, 2021, 02:55 PM UTC

    This incident has been resolved.

  5. postmortem Apr 07, 2021, 04:36 PM UTC

    The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the lack of availability of the Digital Health Programs Care Team and Consumer UI applications on April 01, 2021. # Event Description At approximately 15:21 MDT on April 01, 2021, Healthwise’s Security Operations team was alerted to the unavailability of the Digital Health Programs applications. Healthwise's cloud hosting provider \(Microsoft Azure\) began experiencing an "anomalous surge in DNS queries from across the globe" causing decreased availability of Microsoft Azure's DNS service. Microsoft Azure rerouted DNS traffic, and functionality was restored at approximately 16:57 MDT. The total duration of the incident was approximately 96 minutes. # Findings and Root Cause Based on the investigation conducted, Healthwise determined the following findings regarding this event: * decreased availability of Microsoft Azure's DNS system, due to unexpected increase in DNS queries * Microsoft Azure's RCA at [https://status.azure.com/status/history/](https://status.azure.com/status/history/) under the heading "RCA - DNS issue impacting multiple Microsoft services \(Tracking ID GVY5-TZZ\)" indicated their responsibility for the issue # Corrective Action Healthwise is not recommending or implementing any corrective actions at this time.