Tideways experienced a minor incident on June 14, 2026 affecting Ingest / Data Collection, lasting 3h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 14, 2026, 07:14 PM UTC
We are investigating slow processing of historical data aggregation job.
- identified Jun 14, 2026, 09:13 PM UTC
The issue was identified as not locking the processing for the same application/project, which lead to all workers re-processing the same project which was taking a few minutes and then breaking with an out of memory error. Multiple fixes are now in place and the queue for historical data is processing to close the gap as soon as possible.
- monitoring Jun 14, 2026, 09:39 PM UTC
Processing queue has dropped from over 60k tasks to 39k tasks now. All historical data for June 12th have been processed and now those from June 13th are being processed as fast as possible.
- resolved Jun 14, 2026, 10:19 PM UTC
Processing will be up to date in the next 60-90 minutes, before processing for June 14th gets scheduled. This restores equilibrium in processing historical data.