Iterable Outage History

Started Jul 28, 2025, 10:21 AM UTC· resolved Jul 28, 2025, 10:48 AM UTC

1. investigating Jul 28, 2025, 10:21 AM UTC

   Beginning at 2:30 AM PDT, Iterable Engineering detected an issue with our C18 clusters leading to delays on user updates, list uploads, user deletion, and event processing. Additionally, there were failures on API calls to endpoints such as event/track for a period of time. Next update at 3:45 AM PDT or sooner.

2. monitoring Jul 28, 2025, 10:39 AM UTC

   Ingestion lag for C18 is recovering and success rates for all services are back to normal. We will continue to monitor the cluster until the ingestion backlog has fully cleared. Next update at 4:30 AM PDT or sooner.

3. monitoring Jul 28, 2025, 10:39 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Jul 28, 2025, 10:48 AM UTC

   The system has finished working on backlog of data and the data is caught up. All systems are operational and have no issues. Engineering will continue to monitor the cluster performance.

Read the full incident report →

Started Jul 03, 2025, 02:27 PM UTC· resolved Jul 03, 2025, 04:29 PM UTC

1. investigating Jul 03, 2025, 02:27 PM UTC

   Around 2:16 AM PDT, we detected delays on journey processing. Current impact to our platform is stuck users and slow email sends, we are confirming other impact. Our team is actively working to resolve this as soon as possible. Next update will be provided at 8:30am PDT.

2. monitoring Jul 03, 2025, 02:46 PM UTC

   We have deployed a fix to production at 7:30 AM PDT. Engineering continues to monitor effectiveness of the fix.

3. resolved Jul 03, 2025, 04:29 PM UTC

   This incident is resolved and the backlog of processing has recovered. After further analysis engineering has provided an update that journey recovery started 6:30 to 7AM PST.

Read the full incident report →

Started Jun 12, 2025, 07:17 PM UTC· resolved Jun 12, 2025, 09:16 PM UTC

1. identified Jun 12, 2025, 07:17 PM UTC

   Starting around 11:30 AM PT, Iterable Engineering detected a growing lag across clusters causing delays with user updates, list uploads, user deletion, and event processing. Engineering has identified the cause related to an upstream network provider and is working to remediate these delays. Next update at 1:00 PM PT or sooner

2. identified Jun 12, 2025, 07:59 PM UTC

   We are still seeing ingestion lag across all clusters. No data is being dropped; however, there are delays in user updates, list uploads, user deletion, and event processing, which may also impact journey triggers. Next update at 2:00 PM PT or sooner

3. monitoring Jun 12, 2025, 08:45 PM UTC

   We are seeing ingestion lag recovering across all clusters and catching up on the remaining backlog. The team is continuing to monitor. Next update at 2:15 PM PT or sooner

4. resolved Jun 12, 2025, 09:16 PM UTC

   Ingestion lag has recovered across all clusters. This incident is now resolved.

Read the full incident report →

Started Jun 06, 2025, 04:00 AM UTC· resolved Jun 06, 2025, 04:15 AM UTC

1. resolved Jun 06, 2025, 05:23 PM UTC

   US Data Center Sends Cluster Outage Summary: Between 12:14 AM and 12:55 AM EDT, our US Data Center's sends cluster experienced a critical outage, effectively preventing message delivery. All publishes during this period were safely backed up to our secondary queues. Resolution: We initiated recovery efforts at 12:55 AM EDT, working through the accumulated backlog of unsent messages. During this time, sends were delayed but actively processing. By approximately 1:48 AM EDT, the backlog was cleared, and the sends cluster in the US Data Center was fully recovered. Current Status: We are fully recovered and do not anticipate any further issues related to this incident. All messages are now being sent without delay.

Read the full incident report →

Started May 20, 2025, 02:52 PM UTC· resolved May 20, 2025, 07:02 PM UTC

1. investigating May 20, 2025, 02:52 PM UTC

   Starting at around 9 pm PT 5/19, cluster 22 began experiencing delays delays on user updates, list uploads, user deletion, and event processing. Customers could experience delays of up to 6 hours currently. The engineering team is currently investigating and working to remediate these delays. Next update at 9:30 am PT or sooner.

2. investigating May 20, 2025, 03:20 PM UTC

   We are continuing to investigate this issue.

3. monitoring May 20, 2025, 04:34 PM UTC

   Our engineering team has rolled out a fix to remediate the delays seen on c22 for user updates, list uploads, user deletion, and event processing. The system is currently working through the backlog of data and we should see it drained in a couple of hours. Engineering will continue to monitor the cluster performance until the backlog is completely drained. Next update at 11:30 am PT or sooner.

4. resolved May 20, 2025, 07:02 PM UTC

   The system has finished working on backlog of data and the data is caught up. All systems are operational and have no issues. Engineering will continue to monitor the cluster performance.

Read the full incident report →

Started May 19, 2025, 09:42 PM UTC· resolved May 19, 2025, 10:28 PM UTC

1. investigating May 19, 2025, 09:42 PM UTC

   Mailgun is reporting degraded performance across their application https://status.mailgun.com/. We are monitoring mailgun and their performance.

2. monitoring May 19, 2025, 09:43 PM UTC

   We are monitoring the affect of the mailgun degradation

3. monitoring May 19, 2025, 09:54 PM UTC

   Mailgun has implemented a fix and is in Monitoring status

4. resolved May 19, 2025, 10:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 23, 2025, 06:50 AM UTC· resolved Apr 24, 2025, 11:17 PM UTC

1. investigating Apr 23, 2025, 06:50 AM UTC

   All queries to Cluster C120 are failing as there was a problem in cluster formation. The problem started around 4/22/2025 19:10 pm PDT. Any campaign sent during this time would have been impacted. Ingestion lag is currently building up. The team is restoring the data into an alternate cluster to bring back all organizations into a working state. This can take several hours. The ingestion lag will deplete after the organizations are restored. No data should be lost. Next update at 4/23/2025 6 am PDT.

2. investigating Apr 23, 2025, 06:51 AM UTC

   We are continuing to investigate this issue.

3. investigating Apr 23, 2025, 02:33 PM UTC

   The team has finished restoring data from C120 to a new cluster. Engineering team is working on migrating the in-flight data. This can take few hours, and we will update once we receive a time frame. The ingestion lag will deplete once the data for the organizations are restored. No data should be lost, however event triggered journeys, exports to Snowflake, and campaign sends will be delayed. Next update will be at 4/23/2025 9 am PDT.

4. identified Apr 23, 2025, 04:04 PM UTC

   The team is still working on a cutover to the new cluster. Once the changes propagate to all of the services we should start to see the ingestion lag deplete. Once cutover is complete Blast campaigns should resume. Next update will be at 4/23/2025 10 am PDT.

5. identified Apr 23, 2025, 05:17 PM UTC

   Iterable's engineering team is continuing their work on the cutover to the new cluster. They are working on finalizing details to make sure we minimize impact as much as possible. Next update will be at 4/23/2025 11 am PDT.

6. monitoring Apr 23, 2025, 06:16 PM UTC

   At this point all services have up-to date changes. Iterable's engineering team have cutover to the new cluster. The impacted customers are in the process of being notified. We are actively monitoring. Next update will be at 4/23/2025 around 12:15 pm PDT.

7. monitoring Apr 23, 2025, 07:23 PM UTC

   Engineering successfully completed the cluster cutover at 11:03 AM PDT. Ingestion for customers previously on Cluster 120 is now flowing normally through the new Cluster 141. We’ve also confirmed that campaign scheduling has resumed and is running successfully on Cluster 141. Next update will be provided by 3:00 PM PDT.

8. resolved Apr 23, 2025, 10:07 PM UTC

   Engineering has confirmed that the cluster is performing as expected. No new issues have occurred since the last update. We also wanted to share that further investigation post-incident has uncovered that, during the impact window, users that were set to start journeys via custom events, system events, and user profile updates did not enter journeys. However, for journeys initiated by scheduled, API calls, and "other journey" actions, the system functioned as expected, and users progressed through their journeys. The incident has now been marked as resolved. If you have questions, please reach out to our Support team.

Read the full incident report →

Started Mar 14, 2025, 04:56 PM UTC· resolved Mar 14, 2025, 09:13 PM UTC

1. investigating Mar 14, 2025, 04:56 PM UTC

   Our engineering team has identified an issue causing ingestion delay for all Clusters 100 and above. We are deploying the fix right now. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:30 AM PDT or sooner.

2. investigating Mar 14, 2025, 05:31 PM UTC

   The engineering team is still working on the fix, and customers on Clusters 100 and above will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 11:00 AM PDT or sooner.

3. identified Mar 14, 2025, 06:07 PM UTC

   Engineering has put in the fix. Some clusters have fully recovered, and we are actively working on the ones that have not. For those customers that are still impacted, you will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 12:30 PM PDT or sooner.

4. monitoring Mar 14, 2025, 07:30 PM UTC

   Engineering has finished pushing the fix. Customers on Clusters 116, 120, and 132 may still see an ingestion lag greater than 1 hour. All other Clusters have recovered. Data was never dropped. The next update is at 3 PM PDT or sooner.

5. resolved Mar 14, 2025, 09:13 PM UTC

   Our engineering team has confirmed at 2:06 PDT that all ingestion should be back to normal and is stable. This incident is now resolved.

Read the full incident report →

Started Mar 12, 2025, 03:45 PM UTC· resolved Mar 18, 2025, 08:11 PM UTC

1. investigating Mar 12, 2025, 03:45 PM UTC

   The engineering team has received reports of customers experiencing degraded performance with campaign sends, scheduled journeys, and exports. They are currently investigating the issue. If you believe you're seeing this impact, please contact [email protected]. The next update will be at 10:30 AM PST.

2. investigating Mar 12, 2025, 05:33 PM UTC

   Investigating - The engineering team is still investigating reports of customers experiencing degraded performance with campaign sends, scheduled journeys, and exports. If you believe you're seeing this impact, please contact [email protected]. The next update will be before 5PM PST today.

3. investigating Mar 12, 2025, 11:49 PM UTC

   The engineering team is continuing to investigate reduced performance with campaign sends, scheduled journeys, and exports. This issue occurs sporadically and does not affect every campaign or journey. If you think you're affected by this issue, please contact [email protected]. We will provide the next update by 10 AM PDT tomorrow or before.

4. investigating Mar 13, 2025, 04:38 PM UTC

   Engineering has continued working through its remediation steps. Currently, it is focusing on testing a fix on certain services individually to help ensure that our customers do not feel a greater impact. We will provide the next update by 12 PM PDT or sooner.

5. investigating Mar 13, 2025, 07:20 PM UTC

   Engineering is continuing to pursue remediation steps. They are still investigating different solutions and attempting to test fixes on certain services individually to help ensure that more significant impact is not felt for our customers. We will provide the next update by 3 PM PDT or sooner.

6. investigating Mar 13, 2025, 09:17 PM UTC

   Engineering continues to work on remediation steps and is testing fixes on individual services to minimize customer impact. The scope of the issue remains limited, affecting less than 1% of calls across our customer base. Customers may experience sporadic errors or failures in the following areas: - Scheduled campaigns: Delays in starting or finishing - Scheduled journey triggers - Partial sends for a subset of scheduled campaigns - Export list calls We appreciate your patience as we work toward a resolution. The next update will be provided by 10 AM PDT tomorrow or sooner if significant progress is made.

7. investigating Mar 14, 2025, 04:26 PM UTC

   Engineering has implemented remediation steps, but they have not resolved the issue. The team is actively exploring alternative solutions and evaluating their effectiveness. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We appreciate your patience and will provide the next update by 1 PM PDT or sooner if we have significant updates

8. investigating Mar 14, 2025, 08:17 PM UTC

   Engineering continues to pursue different fixes. Performance for our customers has been partially remediated and improved. Customers should be experiencing a reduction in delays in campaign sends, journey triggers, and processing. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update by 5 PM PDT or sooner if we have significant updates.

9. investigating Mar 14, 2025, 11:05 PM UTC

   Engineering is still investigating a complete fix, but is still seeing performance improvements across delays in campaign sends, journey triggers, and processing. We are continuing to work on rolling out improvements and solutions. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update tomorrow by 10 AM PDT or sooner if we have significant updates.

10. investigating Mar 15, 2025, 04:40 PM UTC

    Engineering is still investigating a complete fix, and are monitoring changes we put in last night to see if it mitigates this issue. We are seeing performance improvements across delays in campaign sends, journey triggers, and processing. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update before 3 PM PDT or sooner if we have significant updates.

11. investigating Mar 15, 2025, 09:44 PM UTC

    Engineering is actively monitoring the fix implemented last night. While initial metrics show signs of improvement, we are still assessing whether delays in campaign sends, journey triggers, and processing have been fully resolved. To ensure a thorough resolution, we will continue tracking the impact of this fix while exploring additional solutions over the weekend. The scope of the issue remains unchanged, affecting fewer than 1% of API calls across our customer base. Our next update will be provided by 3 PM PDT tomorrow, or sooner if we have significant developments.

12. investigating Mar 16, 2025, 10:05 PM UTC

    Engineering continues to monitor the changes we made this weekend. So far, our monitoring has indicated significant performance improvements. The next step is to observe and confirm that the changes have mitigated the delays in campaign sends, journey triggers, and processing after our system has received increased traffic. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update tomorrow before 10 AM PDT or sooner if we have significant updates.

13. monitoring Mar 17, 2025, 04:48 PM UTC

    Engineering has reviewed the changes it pushed through this weekend and has not seen any delays in scheduled campaigns, scheduled journey triggers, partial sends for a subset of scheduled campaigns, and export list calls. We are currently monitoring the fix across all our systems to ensure we did not miss any services. While the fix is in, it might also take some time for the changes to fully propagate. We will provide another update by 3PM PDT or sooner if we have significant updates.

14. monitoring Mar 17, 2025, 09:57 PM UTC

    Engineering is no longer seeing any delays in scheduled campaigns, scheduled journey triggers, partial sends for a subset of scheduled campaigns, and export list calls. We will continue to monitor this fix across all our systems and ensure there are no errors for the next day. If we continue to see no errors, we will resolve this issue tomorrow at 12 PM PDT. We will provide another update by tomorrow at 12 PM PDT or sooner if we have significant updates.

15. resolved Mar 18, 2025, 08:11 PM UTC

    Our Engineering team has confirmed at 12:43PM PDT that all scheduled campaigns, scheduled journey triggers, sends for scheduled campaigns, and export list calls are behaving normally and are stable. This incident is now resolved.

Read the full incident report →

Started Mar 07, 2025, 09:57 AM UTC· resolved Mar 07, 2025, 06:35 PM UTC

1. investigating Mar 07, 2025, 09:57 AM UTC

   We are currently investigating an issue affecting Journeys where fetching, creation and saving of journeys is not working correctly. Customers may experience delays or unexpected behavior when using Journeys. Our Engineering team is actively working to identify the root cause. We will provide the next update in 60 minutes or sooner if more information becomes available.

2. monitoring Mar 07, 2025, 10:32 AM UTC

   A fix has been deployed and journey function is operating normally. Moving to monitoring state.

3. resolved Mar 07, 2025, 06:35 PM UTC

   We do not see any more errors in journey edits.

Read the full incident report →

Started Jan 13, 2025, 10:41 PM UTC· resolved Jan 14, 2025, 01:03 AM UTC

1. identified Jan 13, 2025, 10:41 PM UTC

   Our engineers have identified and are deploying a fix for an issue with ingestion delays. The current delay is roughly 4 hours. This issue only affects customers on cluster 22. You can find your cluster id displayed on the Project Settings page. Next update by 3:30PST.

2. identified Jan 13, 2025, 11:32 PM UTC

   Ingestion backlogs on c22 are shrinking due to the deployed fix. Next update by 4:30PST.

3. monitoring Jan 14, 2025, 12:30 AM UTC

   Ingestion on c22 has fully caught up. Our engineering team is continuing to monitor the situation.

4. resolved Jan 14, 2025, 01:03 AM UTC

   Our engineering team hasn't detected any additional anomalies, and ingestion has been stable. This incident is now resolved.

Read the full incident report →

Started Jan 07, 2025, 07:56 PM UTC· resolved Jan 07, 2025, 10:10 PM UTC

1. investigating Jan 07, 2025, 07:56 PM UTC

   Our on-call engineers are investigating intermittent partial API outages lasting several minutes over the past couple hours. Our monitoring and alerting first detected the issue near 8:53 am PST. Next update by 1PM PST.

2. monitoring Jan 07, 2025, 09:01 PM UTC

   Our engineering team implemented a fix for the issue and have seen no further API outages. We are continuing to monitor the situation.

3. resolved Jan 07, 2025, 10:10 PM UTC

   We have not seen further API outages since the last update. This incident is now resolved.

Read the full incident report →

Started Dec 31, 2024, 02:29 PM UTC· resolved Dec 31, 2024, 06:25 PM UTC

1. identified Dec 31, 2024, 02:29 PM UTC

   Summary: Event-Triggered Journeys Delays Ingesting New Users. The issue originated from errors in the workflow-entrance-trigger pods, causing a significant backlog in processing. There is no impact to Scheduled Journeys and API Triggered Journeys . Actions Taken The workflow-entrance-trigger service was updated to the latest version, and additional pods were scaled up to process the backlog faster. The deployment resolved the issue, and error rates dropped significantly. Current Status The errors we were experiencing have been fixed since 5AM PST, now we're just monitoring the backlog as it drains. For 99% of clients, the backlog has drained completely, there are a few stragglers with small backlogs Next Steps Engineers will continue monitoring error rates and ensure the backlog clears entirely. Follow-up tasks include setting up error rate monitoring and addressing journey-specific issues to prevent recurrence.

2. identified Dec 31, 2024, 04:11 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Dec 31, 2024, 06:25 PM UTC

   This issue is now resolved and the backlog is completely drained. Event trigger journey's as of 7:25 AM PST are back to normal and processing as expected. If you still have any further questions please reach out to [email protected].

Read the full incident report →

Started Nov 25, 2024, 05:07 PM UTC· resolved Nov 25, 2024, 08:18 PM UTC

1. investigating Nov 25, 2024, 05:07 PM UTC

   Intermittent slowness observed on segmentation page. The engineering team is currently investigating and we will update at 9:45am PT.

2. identified Nov 25, 2024, 05:25 PM UTC

   The issue has been identified and fix has been implemented. The engineering team will continue monitoring systems and we will update at 10:15am PT or sooner.

3. monitoring Nov 25, 2024, 05:47 PM UTC

   The issue has been mitigated. The engineering team will continue monitoring systems and we will update at 11:30am PT or sooner.

4. monitoring Nov 25, 2024, 07:06 PM UTC

   The issue has been mitigated. The engineering team continues to monitor systems. If you are still seeing any impact please reach out to [email protected]

5. resolved Nov 25, 2024, 08:18 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Nov 24, 2024, 02:49 PM UTC· resolved Nov 25, 2024, 05:09 PM UTC

1. investigating Nov 24, 2024, 02:49 PM UTC

   Beginning at 5 AM PST we were alerted to a spike in API errors and Web App errors. Our engineering team is investigating. Next update will be at 7:30 AM PST or sooner. If you have questions please reach out to [email protected]

2. investigating Nov 24, 2024, 03:31 PM UTC

   Iterable engineering has put in mitigations and is seeing recovery in the web application and API endpoints. We are continuing to work on remediating delays to sends and journey processing. Next update by 8:30 am PT

3. investigating Nov 24, 2024, 04:25 PM UTC

   We're seeing continued improvements in the web application and API performance. We are still working to resolve delays in sends and webhooks. Next update by 9:00 AM PT or sooner

4. investigating Nov 24, 2024, 05:01 PM UTC

   We have put in further mitigations to restore send speeds and are working through the send backlog. We are continuing to work on recovering webhooks. Next update by 9:30 AM PT or sooner

5. investigating Nov 24, 2024, 05:15 PM UTC

   The Engineering team is investigating web and API errors. Next update by 9:45 AM PT or sooner.

6. investigating Nov 24, 2024, 05:47 PM UTC

   The Engineering team is implementing mitigations to address the intermittent web and API errors as well as send and webhook delays. Next update by 10:15 AM PT or sooner.

7. investigating Nov 24, 2024, 06:16 PM UTC

   The Engineering team is continuing to investigate and implement mitigations to address DB issues causing web and API errors. Sends, journey processing, and webhooks will continue to see delays. Customers may also experience issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 10:45 AM PT or sooner.

8. investigating Nov 24, 2024, 06:49 PM UTC

   The Engineering team is continuing to investigate and implement mitigations to address DB issues causing web and API errors. Sends, journey processing, and webhooks will continue to see delays. Customers may also experience issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 11:15 AM PT or sooner.

9. investigating Nov 24, 2024, 07:16 PM UTC

   The Engineering team is continuing to investigate and implement mitigations to address DB issues. Web errors and journey latency are going down and send rate is going up. Customers may still experience degradation and issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 11:45 AM PT or sooner.

10. identified Nov 24, 2024, 07:49 PM UTC

    The Engineering team is continuing to address DB issues. We are slowly bringing up the scaled down instances of various services. Web errors and journey latency are going down and send rate is going up. Customers may still experience issue with web app failing to fully load and an inability to switch projects. Next update by 12:15 PM PT or sooner.

11. identified Nov 24, 2024, 08:21 PM UTC

    The issue is now identified. The engineering team is slowly bringing up the scaled down instances of various services. Customers should begin seeing improvements in access to the web app, and message send rates are beginning to recover. All services except webhooks are scaled up now. We should continue to see improvements across the app as we catch up on the messaging backlog. Next update by 12:45 PM PT or sooner.

12. identified Nov 24, 2024, 08:50 PM UTC

    The issue is identified. The engineering team continues to scale up instances of various services. Customers should begin seeing improvements in access to the web app, and message send rates are beginning to recover. We should continue to see improvements across the app as we catch up on the messaging backlog. Next update by 1:15 PM PT or sooner.

13. identified Nov 24, 2024, 09:20 PM UTC

    The issue is identified. The engineering team continues to scale up instances of various services. Customers should continue seeing improvements in access to the web app, and message send rates continue to recover. We continue to see improvements across the app as we catch up on the messaging backlog. Next update by 2:15PM PT or sooner.

14. monitoring Nov 24, 2024, 10:23 PM UTC

    The engineering team has completed the instances scale up of various services. Access to the web app, and message send rates are back to normal processing. We continue to see improvements across the app as we catch up on the messaging backlog. Most systems are caught up with processing. Next update by 3:15PM PT or sooner.

15. monitoring Nov 24, 2024, 11:31 PM UTC

    Engineering team has confirmed that we continue to process at normal rates. Our processing of messaging backlog has caught up. Engineering is continuing to monitor our systems, if you are still seeing any impact please reach out to [email protected]

16. monitoring Nov 25, 2024, 11:06 AM UTC

    We are continuing to monitor for any further issues.

17. monitoring Nov 25, 2024, 02:18 PM UTC

    The Engineering team continues to monitor system performance. There have been no performance degradations. Next update by 12 pm PT or sooner. If you are still seeing any impact please reach out to [email protected]

18. monitoring Nov 25, 2024, 05:08 PM UTC

    This incident has been resolved.

19. resolved Nov 25, 2024, 05:09 PM UTC

    This incident has been resolved.

Read the full incident report →

Started Nov 22, 2024, 03:15 PM UTC· resolved Nov 22, 2024, 08:20 PM UTC

1. investigating Nov 22, 2024, 03:15 PM UTC

   Iterable engineering became alerted to high error rates on web-app this morning around 6:00 a.m. PT. After investigating the this issue it appears that these spikes could be happening at the top of each hour as higher volumes of campaigns are scheduled to be sent out. The first time frame of impact was from 6:02 - 6:11am PT and then again at 7am PT. Customers could be seeing a slowness in the web UI. Sends processing, journey processing, and data ingestion are not currently impacted. Next update at 8 am PT or sooner.

2. monitoring Nov 22, 2024, 04:08 PM UTC

   Iterable engineering has mitigated the errors and slowdown affecting the web UI and API endpoints at the top of the hour. The last one occurred between 7:02 - 7:12 am PT. We are continuing to monitor. Next update by 9 am PT

3. monitoring Nov 22, 2024, 05:11 PM UTC

   There have been no further application degradations, and the engineering team will continue monitoring through the morning. Next update by 12 pm PT or sooner

4. resolved Nov 22, 2024, 08:20 PM UTC

   There have been no further application degradations and the platform looks stable. This will be our last update as we are marking this incident as resolved. Engineering team will continue to monitor closely.

Read the full incident report →

Started Nov 20, 2024, 09:56 PM UTC· resolved Nov 21, 2024, 05:37 PM UTC

1. investigating Nov 20, 2024, 09:56 PM UTC

   We are currently experiencing errors and slowness in accessing the Iterable platform. This is only impacting the Web Interface and is not impacting sends, journey processing. The engineering team is currently investigating. Next update by 3pm PT.

2. monitoring Nov 20, 2024, 11:13 PM UTC

   Engineering has confirmed that the errors and slowness were from 1:30-1:50 PM PT. We have recovered since then, and we're continuing to monitor and work on further preventative measures. Next update by 4 pm PT.

3. identified Nov 21, 2024, 12:10 AM UTC

   The engineering team has identified a fix for the intermittent degradations and is working on a remediation. We're continuing to monitor and add further preventative measures. Next update by 5 pm PT.

4. monitoring Nov 21, 2024, 01:12 AM UTC

   The engineering team has deployed a fix for the intermittent degradations. We are continuing to monitor. If you are still seeing issues with platform slowness, please contact Iterable support. Iterable campaign sends and journey processing were not impacted during this incident. Next update at 7pm PT.

5. monitoring Nov 21, 2024, 03:08 AM UTC

   There have been no further application degradations, and the engineering team will continue monitoring through the morning. Next update by 9am PT

6. resolved Nov 21, 2024, 05:37 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 20, 2024, 03:34 PM UTC· resolved Nov 20, 2024, 08:33 PM UTC

1. investigating Nov 20, 2024, 03:34 PM UTC

   From 4:45 AM to 5:45 AM PT, we saw errors and slowness reaching app.iterable.com. Accessing all areas of the platform may have been met with slow loads, but sends and logins are not affected. The engineering team is currently investigating and we will update at 8:30am PT.

2. monitoring Nov 20, 2024, 04:51 PM UTC

   Iterable Engineering has confirmed that we've recovered from the errors that were previously noted as reaching app.iterable.com. Users should be seeing platform speeds return to normal. Engineering is continuing to monitor and put preventative measures in place, but if you are still seeing any impact with Iterable pages loading slowly please reach out to [email protected]

3. resolved Nov 20, 2024, 08:33 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 14, 2024, 01:00 PM UTC· resolved Nov 14, 2024, 03:30 PM UTC

1. resolved Nov 14, 2024, 05:45 PM UTC

   Today, November 14th around 5:00 am PST, Iterable was alerted to an issue where inApp campaigns were not rendering handlebars correctly. Iterable engineers began investigating the issue and identified the cause of the issue had started around 10:00 am PST on November 13th. Once they deployed their fix today at 7:30 am PST handlebars began rendering correctly again for all in App campaigns. During this impact window customers may have also seen issues with in App messages not sending / being fetched correctly in regards to dynamic content not being available. If you have any additional questions or think you may have been impacted please reach out to [email protected]

Read the full incident report →

Started Nov 12, 2024, 05:18 PM UTC· resolved Nov 12, 2024, 07:24 PM UTC

1. investigating Nov 12, 2024, 05:18 PM UTC

   Starting at around 6:30 am PT, Iterable Engineers were notified that lag on our ingestion topics was growing on a subset of clusters. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:00 an PT or Sooner.

2. monitoring Nov 12, 2024, 05:52 PM UTC

   The engineering team has identified the cause and remediated the delays on these subset of clusters. Most impacted ingestion topics have recovered and customers should start to see data being ingested normally again. We are still working through some backlog, so the team will continue to monitor to ensure that these delays don't recur. Next Update at 11:00 am PT or sooner?

3. resolved Nov 12, 2024, 07:24 PM UTC

   Ingestion has returned to normal for all impacted clusters and this issue is now resolved.

Read the full incident report →

Started Oct 25, 2024, 06:48 PM UTC· resolved Oct 26, 2024, 07:35 PM UTC

1. identified Oct 25, 2024, 06:48 PM UTC

   The code fix for the slowness of System webhooks is deployed. Currently the queue is depleting. This could take several hours to deplete the backlog completely. During this time the performance will be degraded. Next update at 6pm PST.

2. monitoring Oct 26, 2024, 12:16 AM UTC

   System webhook backlog is depleted and the incident is resolved.

3. resolved Oct 26, 2024, 07:35 PM UTC

   This incident has been resolved.

Read the full incident report →