Focsec incident

Degraded API performance

Minor Resolved View vendor source →

Focsec experienced a minor incident on May 10, 2023, lasting 11h 56m. The incident has been resolved; the full update timeline is below.

Started
May 10, 2023, 01:04 AM UTC
Resolved
May 10, 2023, 01:01 PM UTC
Duration
11h 56m
Detected by Pingoru
May 10, 2023, 01:04 AM UTC

Update timeline

  1. investigating May 10, 2023, 01:04 AM UTC

    We are seeing high latency response times on the API. We are actively investigating this issue..

  2. identified May 10, 2023, 02:06 AM UTC

    This issue has been identified and we are working on a solution. Most API response times are back to normal, but we are still seeing some high latency spikes. Our team is working on additional fixed to fully restore performance. We are also seeing a small number of API errors (502) related to this issue..

  3. monitoring May 10, 2023, 02:20 AM UTC

    We are seeing API response times further improving and you should see normal performance on all API endpoints. We continue to monitor our systems for any latency spikes. If you are still seeing slow response times, please contact technical support at [[email protected]](mailto:[email protected]).

  4. resolved May 10, 2023, 01:01 PM UTC

    This issue has been resolved and API response times are fully back to expected levels. We apologize for any inconvenience this incident may have caused The high latency was caused a combination of networking issues on our hosting infrastructure and a unusual high number of complex queries being executed in our database. We have provisioned more computing capacity for our database and further improved our auto-scaling capabilities to avoid this issue in the future..