Empist incident

Degraded Cloud Performance due to server hosts rebooting and failing over

Minor Resolved View vendor source →

Empist experienced a minor incident on February 27, 2017, lasting 4h 6m. The incident has been resolved; the full update timeline is below.

Started
Feb 27, 2017, 04:48 PM UTC
Resolved
Feb 27, 2017, 08:55 PM UTC
Duration
4h 6m
Detected by Pingoru
Feb 27, 2017, 04:48 PM UTC

Update timeline

  1. identified Feb 27, 2017, 04:48 PM UTC

    We observed several Purple Screens today on servers that have caused hosts to fail over and reboot. This has resulted in degraded performance for some users as they get disconnected from their cloud sessions when the hosts are failing over. We have begun deploying emergency patches that are addressing the issues. However, we have an ETR of an hour before all hosts can be fully patched. During this time, users can expect intermittent connectivity. We will advise once all patches have been successfully completed (ETR is 11:45 am CT) and connectivity has returned to normal.

  2. monitoring Feb 27, 2017, 06:58 PM UTC

    We successfully deployed emergency patches to all server hosts that was completed at 11:45 am CT. This resulted in a return to full cloud performance. Since then, we have been monitoring for any other issues and have not detected anything. We will continue monitoring for the next couple hours before setting the status to resolved.

  3. resolved Feb 27, 2017, 08:55 PM UTC

    We have been monitoring the server hosts for the pasts 2 hours since deploying the emergency patches and there have been no further issues and have performed at optimum performance levels during that time. We are closing this issue as resolved.