Trackunit incident

Data delays for Trackunit assets

Minor Resolved View vendor source →

Trackunit experienced a minor incident on July 22, 2026 affecting Trackunit Manager and Trackunit Go and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.

Started
Jul 22, 2026, 05:58 AM UTC
Resolved
Jul 23, 2026, 06:15 AM UTC
Duration
1d
Detected by Pingoru
Jul 22, 2026, 05:58 AM UTC

Affected components

Trackunit ManagerTrackunit GoTrackunit OnTrackunit APIStreaming APITrackunit Verify

Update timeline

  1. investigating Jul 22, 2026, 05:58 AM UTC

    Due to an Trackunit infrastructure issue, some customers can experience data delays in Trackunit Manager, APIs, Go, and On. Delay is around 1 hour.

  2. investigating Jul 22, 2026, 07:08 AM UTC

    We are continuing to investigate this issue.

  3. investigating Jul 22, 2026, 07:36 AM UTC

    We are continuing to investigate this issue. Delay is around 1 hour 10 minutes.

  4. identified Jul 22, 2026, 07:58 AM UTC

    The issue has been identified. Work is in progress to fix the issue.

  5. monitoring Jul 22, 2026, 09:26 AM UTC

    A fix has been implemented and we are monitoring the results. Delay is around 1 hour 30 minutes.

  6. monitoring Jul 22, 2026, 11:01 AM UTC

    We are continuing to monitor for any further issues.

  7. monitoring Jul 22, 2026, 11:08 AM UTC

    We are still seeing several hours of delays for updating telematics data in Manager, apps and APIs. Streaming APIs are almost up-to-date.

  8. monitoring Jul 22, 2026, 12:44 PM UTC

    We are below 30 minutes of delays in the pipelines. We hope to be fully up-to-date within 15 minutes.

  9. identified Jul 22, 2026, 01:38 PM UTC

    We have identified another issue in the pipeline and we are working on a fix.

  10. monitoring Jul 22, 2026, 02:42 PM UTC

    We are monitoring a fix and making small adjustments.

  11. monitoring Jul 22, 2026, 04:21 PM UTC

    We are continuing to monitor the situation.

  12. monitoring Jul 22, 2026, 06:51 PM UTC

    The updates are progressing. In 30 minutes, we should have new updates.

  13. monitoring Jul 22, 2026, 08:01 PM UTC

    We are seeing improvements, still monitoring the situation.

  14. monitoring Jul 22, 2026, 10:07 PM UTC

    The fixes previously implemented are working as expected. Since our systems are working through a large backlog of data we still see delays in data ingestion. We will monitor through the night and update the status page tomorrow morning CEST.

  15. monitoring Jul 23, 2026, 06:15 AM UTC

    The infrastructure issue has been resolved, and Trackunit services should now be operating normally.

  16. resolved Jul 23, 2026, 06:15 AM UTC

    This incident has been resolved.

  17. postmortem Jul 29, 2026, 08:43 AM UTC

    # Incident post mortem: Delayed Data Processing, 22–23 July 2026 **Status:** Resolved **Impact:** Delayed data updates. No data was lost. **Duration of impact:** Approx. 22 July 05:40 UTC to 23 July 06:15 UTC. Data delays varied throughout this period. They were not constant, and not all data was delayed at all times. ## What happened For roughly a day, data arriving from machines and connected devices took longer than normal to appear in the platform. The size of the delay varied over the period: at times data was close to up to date, while during the worst periods information that would usually be visible within minutes was delayed by up to about 1.5 hours. The platform itself stayed online and available throughout. The effect was that the data users were looking at was not as current as it should have been. We apologise for the disruption this caused. No data was lost. All information sent to us during the incident was stored safely and processed once capacity was restored. Once the backlog cleared, every affected data point was available in the platform, including for the period of the delay. ## Timeline of events \(UTC\) | Time | Event | | --- | --- | | 22 July, ~00:00 | Processing capacity begins to degrade. | | 22 July, 05:39 | Monitoring triggers an alert to our on-call engineer, who begins investigating. An incident is declared. | | 22 July, 06:00–07:30 | Investigation works through several possible causes and narrows the problem to a capacity limitation in the systems that move data through the platform. First corrective measures are applied. | | 22 July, 07:37 | Delays persist after the first corrective measures. The incident is escalated to our highest severity level. | | 22 July, 09:00–12:00 | Capacity of the affected systems is increased. Delays begin to reduce and backlogs start clearing. | | 22 July, 12:00–20:00 | Further capacity and configuration changes are rolled out in stages. Some of these surface secondary issues that are also resolved. | | 22 July, 22:14 | Systems confirmed stable and delays reducing steadily. The remaining backlog continues to clear. | | 23 July, 05:18 | All data flows confirmed fully caught up. Delays back to normal levels across the platform. | | 23 July, 06:16 | Incident resolved after a period of monitoring with no recurrence. | ## What we did to resolve it Monitoring triggered an alert to our on-call engineer, who began investigating and declared an incident. The initial symptoms pointed in several directions, and the first corrective measures did not resolve the problem. The incident was escalated to our highest severity level mid-morning to bring in additional teams. Once the capacity constraint was identified, we increased the capacity of the affected systems by upgrading the underlying infrastructure and expanding the number of systems processing data in parallel. These changes were rolled out in stages and verified as we went. Some of them surfaced secondary issues that were resolved before throughput fully recovered. The accumulated backlog was then cleared in a controlled sequence, with checks at each step that the data was complete and correct. Total time from alert to full resolution was approximately 24 hours, without data loss, and with a varying delay that generally decreased over that period as capacity was added and data queues were processed. ## Root cause The data-handling layer of our platform reached the performance limit of the infrastructure it was running on. Two factors combined created the root cause for this incident: * The storage and compute capacity supporting this layer had less performance headroom than the current load required. Due to a spike in load over time, It had not been upgraded in line with load growth. * At the same time, an internal process generated a much larger volume of data traffic than expected, which consumed a significant share of the remaining capacity. The traffic spike was the trigger; the limited headroom was the underlying cause. Once the system fell behind, it could not recover without additional capacity being added. ## What we are doing to prevent recurrence | Area | Action | | --- | --- | | **Capacity** | The infrastructure supporting this part of the platform has been upgraded to higher performance levels with meaningfully more headroom than current load requires. This was completed during the incident and is permanent. | | **Detection** | Our monitoring alerted us once data was already delayed. We are adding monitoring of capacity headroom so that a developing constraint is detected before it causes data delays. | | **Safeguards** | We are adding controls to limit how much shared processing capacity any single internal process can consume, so an unexpected spike in one place cannot degrade the wider platform. | | **Capacity planning** | We are reviewing headroom across the platform's other data-processing components and establishing a regular review so capacity keeps pace with load growth. | | **Response** | We are reviewing our diagnostic tooling and runbooks for this part of the platform to reduce the time needed to identify this class of problem. | All of the above are tracked internally with named owners and due dates.