Hevo incident

Degraded performance in US cluster

Minor Resolved View vendor source →

Hevo experienced a minor incident on March 19, 2025 affecting UI Console and Data Pipelines, lasting 1h 22m. The incident has been resolved; the full update timeline is below.

Started
Mar 19, 2025, 09:28 AM UTC
Resolved
Mar 19, 2025, 10:51 AM UTC
Duration
1h 22m
Detected by Pingoru
Mar 19, 2025, 09:28 AM UTC

Affected components

UI ConsoleData Pipelines

Update timeline

  1. investigating Mar 19, 2025, 09:28 AM UTC

    We are currently investigating the issue. The workspaces are not accessible to users in the US region.

  2. identified Mar 19, 2025, 09:34 AM UTC

    The issue has been identified, and a fix is being implemented. During this time, the US cluster will be in read-only mode temporarily.

  3. monitoring Mar 19, 2025, 09:47 AM UTC

    The US cluster has been moved out of read-only mode, and we are monitoring the changes. You would observe no ingestion/load for the pipelines during the time when the cluster was in read-only mode.

  4. resolved Mar 19, 2025, 10:51 AM UTC

    The incident has been resolved. The issue occurred due to unhealthy API nodes which resolved on restarting the service. All services should now be functioning as expected.