Our Sunday Visitor incident

Hub is responding slowly and some user are having problems getting logged in

Major Resolved View vendor source →

Our Sunday Visitor experienced a major incident on October 26, 2020 affecting API, lasting 1h 15m. The incident has been resolved; the full update timeline is below.

Started
Oct 26, 2020, 02:35 PM UTC
Resolved
Oct 26, 2020, 03:50 PM UTC
Duration
1h 15m
Detected by Pingoru
Oct 26, 2020, 02:35 PM UTC

Affected components

API

Update timeline

  1. investigating Oct 26, 2020, 05:08 PM UTC

    We are currently investigating this issue.

  2. identified Oct 26, 2020, 05:09 PM UTC

    There appears to be a database query that is causing an issue with both the API servers and the database.

  3. monitoring Oct 26, 2020, 05:14 PM UTC

    The query plan for the bad query has been cleared and sessions still processing the query have been stopped. The load on the database server has dropped. Monitoring application to see if it returns to normal status.

  4. monitoring Oct 26, 2020, 05:16 PM UTC

    We continued to see issues with logging in. We are restarting the API to clear any left over issues from the bad database query.

  5. resolved Oct 26, 2020, 05:17 PM UTC

    The API is responding normally and users are able to login.