Peakon incident

Elevated error levels on survey and dashboard

Minor Resolved View vendor source →

Peakon experienced a minor incident on March 27, 2020 affecting Dashboard and Survey and 1 more component, lasting 2h 57m. The incident has been resolved; the full update timeline is below.

Started
Mar 27, 2020, 08:18 AM UTC
Resolved
Mar 27, 2020, 11:16 AM UTC
Duration
2h 57m
Detected by Pingoru
Mar 27, 2020, 08:18 AM UTC

Affected components

DashboardSurveyAPI

Update timeline

  1. investigating Mar 27, 2020, 08:18 AM UTC

    We are seeing an increased error rate for the Peakon dashboard and survey applications. We are investigating to find the root cause of the issue

  2. monitoring Mar 27, 2020, 09:02 AM UTC

    The increased error rate has returned to normal. We are investigating a fix for the underlying issues we have found and will continue to monitor the situation.

  3. monitoring Mar 27, 2020, 09:02 AM UTC

    We are continuing to monitor for any further issues.

  4. resolved Mar 27, 2020, 11:16 AM UTC

    We have identified the root cause to be a temporary spike of requests. The web applications are now operating normally. We will investigate how to better handle these spikes for the long-term.