Abusix incident

High load on our main AbuseHQ databases

Minor Resolved View vendor source →

Abusix experienced a minor incident on July 27, 2022 affecting Web Application and REST API and 1 more component, lasting 3h 27m. The incident has been resolved; the full update timeline is below.

Started
Jul 27, 2022, 09:12 AM UTC
Resolved
Jul 27, 2022, 12:40 PM UTC
Duration
3h 27m
Detected by Pingoru
Jul 27, 2022, 09:12 AM UTC

Affected components

Web ApplicationREST APIMail & Event ProcessingRepshare

Update timeline

  1. investigating Jul 27, 2022, 09:12 AM UTC

    We are currently investigating this issue.

  2. investigating Jul 27, 2022, 10:11 AM UTC

    We are still investigating the issue. One of our database nodes is recovering and it takes some time to sync all data.

  3. monitoring Jul 27, 2022, 11:02 AM UTC

    The database node has recovered and we are ramping up backend services to full speed again, while monitoring the performance. Some queues have built up over the last hours so we need to play catch up a bit. Frontend and API performance should be back to normal.

  4. resolved Jul 27, 2022, 12:40 PM UTC

    The issue has been resolved. There are some remaining queues for customers that receive huge batch reports over night, but that's nothing unusual and will be worked through in the next hours. Sorry for the inconveniences this performance degradation might have caused.