Iterable incident

Ingestion Delay on Clusters 100 and Above

Critical Resolved View vendor source →

Iterable experienced a critical incident on March 14, 2025 affecting Global API Ingestion, lasting 4h 16m. The incident has been resolved; the full update timeline is below.

Started
Mar 14, 2025, 04:56 PM UTC
Resolved
Mar 14, 2025, 09:13 PM UTC
Duration
4h 16m
Detected by Pingoru
Mar 14, 2025, 04:56 PM UTC

Affected components

Global API Ingestion

Update timeline

  1. investigating Mar 14, 2025, 04:56 PM UTC

    Our engineering team has identified an issue causing ingestion delay for all Clusters 100 and above. We are deploying the fix right now. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:30 AM PDT or sooner.

  2. investigating Mar 14, 2025, 05:31 PM UTC

    The engineering team is still working on the fix, and customers on Clusters 100 and above will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 11:00 AM PDT or sooner.

  3. identified Mar 14, 2025, 06:07 PM UTC

    Engineering has put in the fix. Some clusters have fully recovered, and we are actively working on the ones that have not. For those customers that are still impacted, you will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 12:30 PM PDT or sooner.

  4. monitoring Mar 14, 2025, 07:30 PM UTC

    Engineering has finished pushing the fix. Customers on Clusters 116, 120, and 132 may still see an ingestion lag greater than 1 hour. All other Clusters have recovered. Data was never dropped. The next update is at 3 PM PDT or sooner.

  5. resolved Mar 14, 2025, 09:13 PM UTC

    Our engineering team has confirmed at 2:06 PDT that all ingestion should be back to normal and is stable. This incident is now resolved.