Explo incident

Increase Query Latency due to Abnormally High Load

Minor Resolved View vendor source →

Explo experienced a minor incident on March 1, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
Mar 01, 2024, 09:46 PM UTC
Resolved
Mar 01, 2024, 08:30 PM UTC
Duration
—
Detected by Pingoru
Mar 01, 2024, 09:46 PM UTC

Update timeline

  1. resolved Mar 01, 2024, 09:46 PM UTC

    Abnormally high load on our dashboards caused increased query latency for all customers. External effect was dashboards timing out or otherwise taking longer than expected to load data

  2. postmortem Mar 01, 2024, 09:46 PM UTC

    Upon investigating our query runner, we found issues with high memory usage under extreme load that exponentially backed up our query queue. We have resolved issues here, and have additionally increased resources to handle higher load on our query runners.