CyberFOX incident

Issues with high latency and errors on primary database

Major Resolved View vendor source →

CyberFOX experienced a major incident on April 13, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
Apr 13, 2022, 05:30 AM UTC
Resolved
Apr 13, 2022, 05:30 AM UTC
Duration
—
Detected by Pingoru
Apr 13, 2022, 05:30 AM UTC

Update timeline

  1. resolved Apr 26, 2022, 03:49 PM UTC

    The issue has been resolved.

  2. postmortem Apr 26, 2022, 03:50 PM UTC

    An index on our database became corrupted which progressively started to affect some of our distributed services. This resulted in performance issues and some partners not being able to access the Admin portal with a browser although approvals via the mobile app, agent functions, technician mode and other functionality remained unimpacted. The Dev team initially troubleshot the issue and endeavored to resolve it without completely bringing all services offline during the day, but this became necessary and last night between 9:45pm and 12:50am EST we took the server side services down so that the indexes could be fixed. During that period, we also took care of some other DB maintenance that we feel potentially may have contributed to the circumstances causing the initial index corruption.