Peakon incident

Delayed background job processing

Minor Resolved View vendor source →

Peakon experienced a minor incident on January 24, 2019 affecting Slack Integration and API, lasting 3h 5m. The incident has been resolved; the full update timeline is below.

Started
Jan 24, 2019, 06:52 AM UTC
Resolved
Jan 24, 2019, 09:57 AM UTC
Duration
3h 5m
Detected by Pingoru
Jan 24, 2019, 06:52 AM UTC

Affected components

Slack IntegrationAPI

Update timeline

  1. investigating Jan 24, 2019, 06:52 AM UTC

    We are seeing increased response time from our primary database, and are stopping processing of background jobs while we investigate the issue. This will delay the delivery of emails and real time notifications.

  2. identified Jan 24, 2019, 07:18 AM UTC

    We have identified an issue with an underlying database index affecting a limited number of queries. We are now processing background jobs at a reduced rate and monitoring performance.

  3. resolved Jan 24, 2019, 09:57 AM UTC

    This issue has been resolved since 08:07 UTC. We continue to investigate the root cause and put in place changes to ensure this does not repeat.