Iterable had 49 outages in the last 2 years totaling 184h 40m of downtime — averaging 2 incidents per month.
There were 49 Iterable outages since July 5, 2024 totaling 184h 40m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are aware of an issue affecting WhatsApp messaging. Some customers may experience delays or failures sending WhatsApp messages. This appears to be caused by an external service disruption, and we are monitoring for updates. We will share another update by 1pm PST or sooner if new information becomes available.
This incident has been resolved.
We are currently investigating an issue involving users being unable to access the Iterable web app. API traffic, Sends, and Journeys already in progress are unaffected. This outage is limited to the Iterable web application itself.
Engineering has identified a fix and is in the process of deploying to both EU and US web environments to resolve the issue. Next update at 5PM PDT or sooner.
Our engineering team is continuing to work toward a full resolution. While we have mostly restored web application access, there may be some pages within the web app that periodically fail to load until the fix is fully rolled out across the entire application. Next update by 5:30PM PDT or sooner.
Our engineering team has completed the fix to the web application in both the US and EU regions. All services are currently functioning normally. Email sends, API traffic, webhooks, and journeys functioned normally throughout this incident. The only impact was being unable to log into the web application and slow page loads. If you continue to experience any issues, please reach out to Support.
We’re currently investigating an issue where Journey pages are not responding. Engineering is actively applying a fix and will continue investigating the root cause. Next update at 4:00 PM PST / 11:00 PM GMT.
Journey pages are loading normally following the fix. Engineering will continue monitoring.
This incident has been resolved.
We are currently experiencing elevated error rates on several API endpoints in our US region, including Bulk Update, Register Device, and Target. We are actively working to resolve this issue and will provide another update within 30 minutes.
We are seeing improved success rates on the Register Device and Target endpoints. Our engineering team is continuing to work on other endpoints including Bulk Update. Next update within 30 minutes.
Our engineering team is continuing to work on improving success rates for all API endpoints including Bulk Update. Next update in 60 minutes.
We are continuing to monitor for any further issues.
This incident has been resolved.
Current status We have identified an issue in our ingestion service that can cause some updates to be delayed. Customer impact Some customers are seeing delays in list uploads and events appearing in Iterable, in some cases taking 30+ minutes to render. Clusters impacted 22,25,16 Rolled back to a previous stable version across all affected clusters and verified the deployments. We are monitoring ingestion lag and key metrics to confirm performance is stabilizing and that no new update loss is occurring. If you are experiencing delayed data visibility or suspect missing updates, please contact our Support team so we can review your specific projects as part of this incident. Next update 1pm PT
We have caught up with ingestion. Currently monitoring our systems.
This incident has been resolved.
We are currently investigating an issue for one of ES clusters - c102 - The cluster is in a red state, leading to very slow ingestion and widespread API 500 errors. Customers are experiencing failed searches and errors on multiple API endpoints. Ingestion rates are near zero, with lag up to 20 minutes and increasing for internal, realtime, and bulk data pipelines.
Engineering is continuing to investigate this issue. At this time there is no improvement yet to any customers on C102. Next Update at 12:15 PDT or sooner.
Engineering has restored C102 back to a "green" state and ingestion and API calls are succeeding again for this cluster. We will continue to monitor until we are fully through the accumulated backlog. Next update at 1 PM PDT or sooner.
We have fully caught up with any backlog on C102 and all queries and activity through C102 are functioning normally. If you continue to experience any issues please contact Support.
We have been alerted to an issue where proofs sends are not being sent as expected as of 9:00 am PT. Iterable Engineers are actively investigating the issue. Next update by 11:00 am PT or sooner.
The engineering team has implemented a fix and proofs are now successfully sending. We will continue to monitor. Next update at 12:00 PST or earlier.
This incident has been resolved.
Email sends for US-hosted customers using SparkPost were delayed due to elevated 5xx errors and increased API latency from SparkPost’s Transmissions API. EU-hosted traffic was not impacted. All customers using ESPs other than Sparkpost were similarly not impacted. SparkPost has indicated the incident is resolved. We’re monitoring outbound email queues to ensure sends are fully processing as expected. If you continue to experience issues, please contact Support.
SparkPost’s API issue has been resolved, and we’re no longer seeing elevated 5xx errors or increased latency. Incident has resolved.
We are currently investigating an issue where users could not access app.iterable.com and received 502 Bad Gateway errors. EU users were not affected.
A fix has been implemented and we are monitoring the results.
Service has fully recovered and all traffic has been restored to app.iterable.com in the US region. The incident lasted from 9:50 am to 10:15 am Pacific Time, impacting both web and API access with elevated 502, 503, and 504 errors. EU region was unaffected.
We experienced a period of delayed sends (including proofs) as well as errors in user updates from 7am to 9am ET. The issue has been resolved. Some delayed messages may still be processing and will be sent shortly. If customers received 500 responses to API requests, they should re-try those calls.
Our engineering team is actively investigating an issue causing elevated error rates and intermittent page loads in the Iterable web app. A subset of users may see error messages or blank pages when attempting to log in or navigate the UI. At this time, API-based message sending and other backend processing appear to be functioning normally. We will continue to post updates here as we identify the root cause and progress toward mitigation. What You Can Do If you encounter an error page, retry after a few minutes or refresh your browser. next update at 1:30 pm PT or sooner
We are continuing to investigate this issue.
All affected web components are functioning as expected, and our monitoring shows error rates and login success back within normal ranges. The Iterable platform is fully operational. If you continue to see errors or unexpected behavior in the web app: Clear your browser’s cache and cookies for app.iterable.com. Fully close the browser and open a new session, then log in again. If problems persist after these steps, please contact Iterable Support via your usual channel
This incident has been resolved.
API Update is experiencing timeouts starting at 11:00 AM PST. The Engineering team is investigating. Next update at 3:00 PM PST.
API Update has recovered. The engineering team will continue to monitor—next update on January 2nd, at 9 AM PST.
API Update is stable. The engineering team will continue to monitor. Next update at 12:00 PM PST.
The engineering team determined that the issue has been resolved.
We are currently experiencing delays in processing certain message sends and exit-trigger workflows. This is due to an upstream outage affecting some of our backend components that handle message publishing.
Upstream issue has been resolved. we will continue monitoring.
All processing is back to normal. We continue to monitor.
This incident has been resolved.
We are currently experiencing a service disruption with our catalog service, which may prevent some customers from sending messages that use templates with catalog data. Messages that do not use catalog data are not impacted. We've identified the root cause and are actively working on a fix. Please note that no messages have been lost. Our system will automatically retry all failed sends once the service has recovered. We will provide another update in 30 minutes, or sooner if the issue is resolved. We apologize for the inconvenience and thank you for your patience.
We have started rolling out the fix to individual machines and are seeing some of our messaging capabilities recover. Some customers may still have a delay in sends that use templates with catalog data, and ingestion delays for updating catalog data. We are continuing to apply the fix across all affected systems and will monitor the situation closely. No messages have been lost, and our system will automatically retry all failed sends once the service has fully recovered. We will provide another update in 30 minutes, or sooner if the issue is resolved. We appreciate your continued patience.
The issue with our catalog service has been fully resolved, and all services are now operating normally. Any delayed sends and catalog updates are being processed currently. Messages that were queued during the disruption will be automatically retried and delivered.
Status: Investigating Summary: We are currently investigating an issue causing 504 Gateway Time-out errors for requests to the /api/export/data.json endpoint affecting customers hosted in our European Data Center (EUDC) ONLY. All other APIs are functional. The issue was first reported around 5:35 pm CEST. Iterable Engineering is currently investigating the issue and working on remediation steps. Next update at 9:25 PM CEST or earlier.
This incident has been resolved.
Status: Resolved Between 3:50 pm and 5:55 pm CEST customers in our EUDC (European Data Center) region may have experienced an increased 5xx error rate, approximately 10-15%, when making api requests. This issue was isolated to Europe; no impact was observed in other regions. If customers have any questions if they were impacted please reach out to [email protected]
We are seeing issues accessing app.iterable.com. We have identified an issue in one of our services and Engineering is working on a fix. Iterable API is not impacted. All campaigns and journeys are processing as expected. Next update will be by 5PM PDT.
We have addressed the issue in our service and app.iterable.com is accessible again. Engineering is monitoring the situation. Next update will be at or before 5:30PM PDT.
Our login service has recovered and app.iterable.com is fully accessible. If you are still having issues, please reach out to support.
We are currently investigating an increase in embedded Messaging skips, which appears to be impacting most customers. Other sends globally are not impacted. Our team is actively working to identify the root cause of this issue. Next Update: We will provide our next update by 2:00 PM PDT.
We are continuing to investigate this issue.
The issue with embedded messages incorrectly being send skipped has been identified and the team is actively working to remediate it. We have confirmed that the impact is only to embedded messages triggered by Journeys. The aforementioned remediation should be completed shortly and we will continue to monitor that no further send skip issues occur once complete. Next update: We will provide our next update at 3:00 PM PDT.
The send skip behavior for journey triggered embedded messages has been resolved. The team is still working on validating that the remediation has restored full functionality. Next update: We will provide our next update at 4:00 PM PDT.
During the validation of the remediation, the team identified that another fix was necessary to fully restore the functionality of journey triggered embedded messages. They are working on testing and releasing the fix. Next update: We will provide our next update at 6:00 PM PDT.
The additional fix has been tested successfully and deployment is in progress. This process will take another 15-30 minutes to rollout to all customers. This will fully restore functionality for journey triggered embedded messaging, which were the only sends impacted. Next up: Our next update will be at or before 7:00 PM PDT.
This incident has been resolved and functionality has been fully restored for journey triggered embedded messaging sends. If more information is needed, please reach out to [email protected]. This will be the final update.
Beginning at 2:30 AM PDT, Iterable Engineering detected an issue with our C18 clusters leading to delays on user updates, list uploads, user deletion, and event processing. Additionally, there were failures on API calls to endpoints such as event/track for a period of time. Next update at 3:45 AM PDT or sooner.
Ingestion lag for C18 is recovering and success rates for all services are back to normal. We will continue to monitor the cluster until the ingestion backlog has fully cleared. Next update at 4:30 AM PDT or sooner.
We are continuing to monitor for any further issues.
The system has finished working on backlog of data and the data is caught up. All systems are operational and have no issues. Engineering will continue to monitor the cluster performance.
Around 2:16 AM PDT, we detected delays on journey processing. Current impact to our platform is stuck users and slow email sends, we are confirming other impact. Our team is actively working to resolve this as soon as possible. Next update will be provided at 8:30am PDT.
We have deployed a fix to production at 7:30 AM PDT. Engineering continues to monitor effectiveness of the fix.
This incident is resolved and the backlog of processing has recovered. After further analysis engineering has provided an update that journey recovery started 6:30 to 7AM PST.
Starting around 11:30 AM PT, Iterable Engineering detected a growing lag across clusters causing delays with user updates, list uploads, user deletion, and event processing. Engineering has identified the cause related to an upstream network provider and is working to remediate these delays. Next update at 1:00 PM PT or sooner
We are still seeing ingestion lag across all clusters. No data is being dropped; however, there are delays in user updates, list uploads, user deletion, and event processing, which may also impact journey triggers. Next update at 2:00 PM PT or sooner
We are seeing ingestion lag recovering across all clusters and catching up on the remaining backlog. The team is continuing to monitor. Next update at 2:15 PM PT or sooner
Ingestion lag has recovered across all clusters. This incident is now resolved.
US Data Center Sends Cluster Outage Summary: Between 12:14 AM and 12:55 AM EDT, our US Data Center's sends cluster experienced a critical outage, effectively preventing message delivery. All publishes during this period were safely backed up to our secondary queues. Resolution: We initiated recovery efforts at 12:55 AM EDT, working through the accumulated backlog of unsent messages. During this time, sends were delayed but actively processing. By approximately 1:48 AM EDT, the backlog was cleared, and the sends cluster in the US Data Center was fully recovered. Current Status: We are fully recovered and do not anticipate any further issues related to this incident. All messages are now being sent without delay.
Starting at around 9 pm PT 5/19, cluster 22 began experiencing delays delays on user updates, list uploads, user deletion, and event processing. Customers could experience delays of up to 6 hours currently. The engineering team is currently investigating and working to remediate these delays. Next update at 9:30 am PT or sooner.
We are continuing to investigate this issue.
Our engineering team has rolled out a fix to remediate the delays seen on c22 for user updates, list uploads, user deletion, and event processing. The system is currently working through the backlog of data and we should see it drained in a couple of hours. Engineering will continue to monitor the cluster performance until the backlog is completely drained. Next update at 11:30 am PT or sooner.
The system has finished working on backlog of data and the data is caught up. All systems are operational and have no issues. Engineering will continue to monitor the cluster performance.