Iterable Outage History

We are monitoring the affect of the mailgun degradation

Mailgun has implemented a fix and is in Monitoring status

Engineering has finished pushing the fix. Customers on Clusters 116, 120, and 132 may still see an ingestion lag greater than 1 hour. All other Clusters have recovered. Data was never dropped. The next update is at 3 PM PDT or sooner.

A fix has been deployed and journey function is operating normally. Moving to monitoring state.

Ingestion on c22 has fully caught up. Our engineering team is continuing to monitor the situation.

Our engineering team implemented a fix for the issue and have seen no further API outages. We are continuing to monitor the situation.

The issue has been mitigated. The engineering team will continue monitoring systems and we will update at 11:30am PT or sooner.

The issue has been mitigated. The engineering team continues to monitor systems. If you are still seeing any impact please reach out to [email protected]

Iterable engineering has mitigated the errors and slowdown affecting the web UI and API endpoints at the top of the hour. The last one occurred between 7:02 - 7:12 am PT. We are continuing to monitor. Next update by 9 am PT

There have been no further application degradations, and the engineering team will continue monitoring through the morning. Next update by 12 pm PT or sooner

Engineering has confirmed that the errors and slowness were from 1:30-1:50 PM PT. We have recovered since then, and we're continuing to monitor and work on further preventative measures. Next update by 4 pm PT.

The engineering team has deployed a fix for the intermittent degradations. We are continuing to monitor. If you are still seeing issues with platform slowness, please contact Iterable support. Iterable campaign sends and journey processing were not impacted during this incident. Next update at 7pm PT.

There have been no further application degradations, and the engineering team will continue monitoring through the morning. Next update by 9am PT

Iterable Engineering has confirmed that we've recovered from the errors that were previously noted as reaching app.iterable.com. Users should be seeing platform speeds return to normal. Engineering is continuing to monitor and put preventative measures in place, but if you are still seeing any impact with Iterable pages loading slowly please reach out to [email protected]

The engineering team has identified the cause and remediated the delays on these subset of clusters. Most impacted ingestion topics have recovered and customers should start to see data being ingested normally again. We are still working through some backlog, so the team will continue to monitor to ensure that these delays don't recur. Next Update at 11:00 am PT or sooner?

System webhook backlog is depleted and the incident is resolved.

Errors with our Embedded Messaging API are not occurring any more. A code fix has been deployed.

Starting around 7:27 AM PST, Iterable engineers began receiving alerts about elevated error rates on a 3rd-party ESP provider's API endpoints. Iterable has been working directly with our 3rd party vendor and they have released a fix and as of 8:07 AM PST we are seeing successful email sends. Customers using this ESP may experience issues impacting email sends, including elevated email campaign send skips during this impact window Iterable Engineer will continue monitoring the issue. If you have any questions please reach out to [email protected]

The fix from our third party vendor has been stable and we're seeing email campaigns are sending successfully again. We are currently working to resend the emails that were impacted during the impact window. Next update will be at 11:00 am pst or sooner.

Engineering is identifying alternative methods to identify affected campaigns and will contact customers to confirm whether redelivery is needed. Additionally, 3rd party vendor recovered as of 8:07 am PT and there are no issues with send. Next update at 12 pm PT.

We are continuing to monitor for any further issues.

Engineering is identifying alternative methods to identify affected campaigns and will contact customers to confirm whether redelivery is needed. Additionally, the 3rd party vendor recovered as of 8:07 am PT, and there are no issues with send. Next update at 12 pm PT.

Engineering is continuing to work on identifying alternative methods to determine affected campaigns. We will be reaching out to impacted customers to confirm whether redelivery is necessary. The 3rd party vendor recovered at 8:07 AM PT, and there are no ongoing issues with email sends.

Engineering has put in the recovery steps, and are seeing ingestion begin to catch-up. Customers might still see some slight delays while we fully recover. We will update next when ingestion has completely caught up.

We're receiving reports from APAC customers of difficulty accessing the Iterable application. The issue appears to be caused by network connectivity between internet providers in APAC. Impacted customers can use a VPN to connect to the Iterable application. The Iterable Engineering team has confirmed there are no issues impacting the operations of the Iterable platform.

We have identified and fixed the problem that halted journey processing. Journeys are now processing through the backlog and are recovering with no data loss. If you triggered a journey during this time, it is not necessary to retrigger it. No action is needed to ensure your journey is processing again.

We believe the issue has been mitigated and that users should see access returning to normal. We are continuing to monitor. If you are still seeing issues, please contact Iterable support.

Web app and API endpoints have completely recovered at this point. However, there are still a subset of customers that may be experiencing an ingestion lag that is currently draining. These customers may still be seeing delays in user updates, event calls, and event triggered journeys. We are continuing to monitor this and will provide our next update at 11 AM PT or sooner.

We are continuing to monitor for any further issues.

As of now we have completely caught up on ingestion lag with all services returning to normal. We will continue to monitor performance with that next update at 12 PM PT or sooner.