Currents incident

Timeouts and missing results for new runs

Minor Resolved View vendor source →

Currents experienced a minor incident on April 28, 2025, lasting —. The incident has been resolved; the full update timeline is below.

Started
Apr 28, 2025, 05:11 PM UTC
Resolved
Apr 28, 2025, 05:11 PM UTC
Duration
Detected by Pingoru
Apr 28, 2025, 05:11 PM UTC

Update timeline

  1. resolved Apr 28, 2025, 05:11 PM UTC

    Type: Incident Duration: 2 hours and 45 minutes Affected Components: Data Ingestion, API - HTTP REST API, API - Dashboard Browsing Apr 28, 17:11:34 GMT+0 - Investigating - We are currently investigating this incident. Apr 28, 18:46:38 GMT+0 - Identified - The issue is caused by an increased memory pressure on an infrastructure component. We are performing an adjustment to it capacity and taking care of the pending tasks. Apr 28, 19:56:08 GMT+0 - Resolved - The system is back to normal. ### Initial Technical Analysis * A sudden increase in ingress traffic caused one of the OLAP DB to throttle new write and read requests * That caused an accumulation of backlog tasks - an autoscaling kicked off but couldn't cope with the backlog * Increasing the infrastructure capacity resolved the issue. ### Impact The issue affected test results reported between April 25 \~4:30pm-7:30pm GMT * runs could be marked as timed out * test results reporting is missing or delayed for affected runs * CI jobs displaying warnings and connectivity error