Iterable experienced a major incident on October 15, 2024 affecting Global Campaign Sends and Global Proof Sends, lasting 5h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Oct 15, 2024, 04:06 PM UTC
Starting around 7:27 AM PST, Iterable engineers began receiving alerts about elevated error rates on a 3rd-party ESP provider's API endpoints. Iterable has been working directly with our 3rd party vendor and they have released a fix and as of 8:07 AM PST we are seeing successful email sends. Customers using this ESP may experience issues impacting email sends, including elevated email campaign send skips during this impact window Iterable Engineer will continue monitoring the issue. If you have any questions please reach out to [email protected]
- monitoring Oct 15, 2024, 04:49 PM UTC
The fix from our third party vendor has been stable and we're seeing email campaigns are sending successfully again. We are currently working to resend the emails that were impacted during the impact window. Next update will be at 11:00 am pst or sooner.
- monitoring Oct 15, 2024, 06:05 PM UTC
Engineering is identifying alternative methods to identify affected campaigns and will contact customers to confirm whether redelivery is needed. Additionally, 3rd party vendor recovered as of 8:07 am PT and there are no issues with send. Next update at 12 pm PT.
- monitoring Oct 15, 2024, 06:11 PM UTC
We are continuing to monitor for any further issues.
- monitoring Oct 15, 2024, 06:12 PM UTC
Engineering is identifying alternative methods to identify affected campaigns and will contact customers to confirm whether redelivery is needed. Additionally, the 3rd party vendor recovered as of 8:07 am PT, and there are no issues with send. Next update at 12 pm PT.
- monitoring Oct 15, 2024, 07:22 PM UTC
Engineering is continuing to work on identifying alternative methods to determine affected campaigns. We will be reaching out to impacted customers to confirm whether redelivery is necessary. The 3rd party vendor recovered at 8:07 AM PT, and there are no ongoing issues with email sends.
- resolved Oct 15, 2024, 09:32 PM UTC
As of 4:16 PM PST, all affected messages have been successfully processed, and the incident is now resolved. Our engineering team has confirmed that all campaigns have been reviewed, and there are no outstanding issues. The 3rd party vendor's recovery at 8:07 AM PT resolved the initial sending issue, and no further delays or interruptions are expected. We appreciate your patience during this time and apologize for any inconvenience caused. If you have any questions or need further assistance, please reach out to our support team at [email protected]