Iterable incident

Spike in web-app errors causing platform degredation

Minor Resolved View vendor source →

Iterable experienced a minor incident on November 22, 2024 affecting Global Web Application, lasting 5h 5m. The incident has been resolved; the full update timeline is below.

Started
Nov 22, 2024, 03:15 PM UTC
Resolved
Nov 22, 2024, 08:20 PM UTC
Duration
5h 5m
Detected by Pingoru
Nov 22, 2024, 03:15 PM UTC

Affected components

Global Web Application

Update timeline

  1. investigating Nov 22, 2024, 03:15 PM UTC

    Iterable engineering became alerted to high error rates on web-app this morning around 6:00 a.m. PT. After investigating the this issue it appears that these spikes could be happening at the top of each hour as higher volumes of campaigns are scheduled to be sent out. The first time frame of impact was from 6:02 - 6:11am PT and then again at 7am PT. Customers could be seeing a slowness in the web UI. Sends processing, journey processing, and data ingestion are not currently impacted. Next update at 8 am PT or sooner.

  2. monitoring Nov 22, 2024, 04:08 PM UTC

    Iterable engineering has mitigated the errors and slowdown affecting the web UI and API endpoints at the top of the hour. The last one occurred between 7:02 - 7:12 am PT. We are continuing to monitor. Next update by 9 am PT

  3. monitoring Nov 22, 2024, 05:11 PM UTC

    There have been no further application degradations, and the engineering team will continue monitoring through the morning. Next update by 12 pm PT or sooner

  4. resolved Nov 22, 2024, 08:20 PM UTC

    There have been no further application degradations and the platform looks stable. This will be our last update as we are marking this incident as resolved. Engineering team will continue to monitor closely.