Hevo incident

Degraded Performance in US cluster

Major Resolved View vendor source →

Hevo experienced a major incident on October 16, 2024 affecting Sources and Destinations and 1 more component, lasting 5h 9m. The incident has been resolved; the full update timeline is below.

Started
Oct 16, 2024, 09:15 PM UTC
Resolved
Oct 17, 2024, 02:25 AM UTC
Duration
5h 9m
Detected by Pingoru
Oct 16, 2024, 09:15 PM UTC

Affected components

SourcesDestinationsData PipelinesModels

Update timeline

  1. investigating Oct 16, 2024, 09:15 PM UTC

    We are currently investigating the issue.

  2. investigating Oct 16, 2024, 09:24 PM UTC

    We are continuing to investigate this issue.

  3. investigating Oct 16, 2024, 09:52 PM UTC

    We have identified the issue and are actively working on a fix. The cluster has been moved to read-only mode which might affect data ingestion for some time. We will provide further updates shortly.

  4. identified Oct 16, 2024, 09:53 PM UTC

    The issue has been identified and a fix is being implemented.

  5. monitoring Oct 16, 2024, 10:50 PM UTC

    A fix has been implemented and we are monitoring the results.

  6. monitoring Oct 16, 2024, 10:51 PM UTC

    The cluster is moved out of read-only and the ingestion should resume soon. We are monitoring the results after deploying a fix.

  7. resolved Oct 17, 2024, 02:25 AM UTC

    This incident has been resolved.