Batch Outage History
Batch is up right nowPingoru has recorded 68 incidents for Batch: 53 in the last 12 months and 12 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 1 of 720 (0%) |
| May 2026 | 5 | 27 of 744 (4%) |
| June 2026 | 2 | 6 of 720 (1%) |
| July 2026 | 6 | 17 of 744 (2%) |
| August 2026 | 1 | 1 of 744 (0%) |
| September 2026 | 4 | 30 of 720 (4%) |
| October 2026 | 6 | 10 of 588 (2%) |
Incidents, newest first
Minor February 21, 2026 Processing Delays for Mobile Profile Updates 3h 33m Email delivery, Push Delivery, SMS Delivery, Data ingestion
Started Feb 21, 2026, 07:43 PM UTC· resolved Feb 21, 2026, 11:17 PM UTC
- investigating Feb 21, 2026, 07:43 PM UTC
We are investigating a performance degradation within our profile processing pipeline. This issue specifically affects updates sent via our mobile SDKs. As of 21/02/2026 at 17:00 CET, processing times have increased, with current delays of up to 3 hours. This latency may result in delayed execution for triggered automations.
- identified Feb 21, 2026, 08:16 PM UTC
We have identified the issue and deployed a fix. The processing backlog is now beginning to clear, and delivery delays are steadily decreasing. We are monitoring the pipeline closely to ensure performance returns to nominal levels.
- monitoring Feb 21, 2026, 09:19 PM UTC
The processing backlog has been steadily decreasing over the last hour. We continue to monitor the situation closely until all updates are processed and real-time performance is restored.
- resolved Feb 21, 2026, 11:17 PM UTC
This incident has been resolved.
Notice February 19, 2026 Email Composer - Image Upload Issue 21m
Started Feb 19, 2026, 09:30 AM UTC· resolved Feb 19, 2026, 09:51 AM UTC
- investigating Feb 19, 2026, 09:30 AM UTC
We are currently experiencing an issue with our email composer provider that prevents image uploads from functioning correctly. The issue has been escalated to the provider, and we are awaiting a fix on their side. We will share further updates as soon as more information becomes available.
- monitoring Feb 19, 2026, 09:38 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 19, 2026, 09:51 AM UTC
This incident has been resolved.
Notice January 15, 2026 CEP Trigger automation delays 1h
Started Jan 15, 2026, 05:00 PM UTC· resolved Jan 15, 2026, 06:00 PM UTC
- resolved Jan 16, 2026, 10:46 AM UTC
Summary: Between 15 January at 18:00 CET and 15 January at 19:00 CET, a small subset of CEP trigger automations was delayed for up to an hour. Only trigger automations using timers and with timers scheduled to fire between 18:00 and 19:00 CET were affected; this represents less than 2% of all trigger automations during this time period. Impact for the affected trigger automations: * Timer firing was delayed by up to one hour. * As a consequence, the delivery of emails, SMS, and push notifications was delayed by up to one hour. Root Cause: some timers were not accepted by our database system due to an edge case in their data, which blocked the entire timer scheduling pipeline. Resolution: a fix was deployed on 15 January at 18:55 CET, which resolved the issue. By 19:00 CET, the backlog was cleared.
Minor January 7, 2026 Trigger automations delays –
Started Jan 07, 2026, 07:54 AM UTC· resolved Jan 07, 2026, 07:54 AM UTC
- resolved Jan 07, 2026, 07:54 AM UTC
Jan 7, 07:54 UTC Resolved - Trigger automations ran late between 07:00 and 08:40 CET. All messages were delivered; none were lost.
Notice December 22, 2025 CEP Campaign Delivery Degradation (Email, Push & SMS) 3h 17m Dashboard, Email delivery, Push Delivery, SMS Delivery
Started Dec 22, 2025, 01:43 PM UTC· resolved Dec 22, 2025, 05:01 PM UTC
- monitoring Dec 22, 2025, 01:43 PM UTC
An incident affected CEP campaign delivery across Email, Push, and SMS between 11:00 and 14:30 CET. * Campaigns: Messages scheduled during this window were not sent and not retried. Analytics will show zero delivery and engagement. If impacted, consider resending your campaign. * Automations: Delays occurred between 11:40 and 12:15 CET. Affected automations were eventually sent, and the displayed analytics are accurate. The issue has been fixed, and we are actively monitoring the platform.
- resolved Dec 22, 2025, 05:01 PM UTC
This incident has been resolved.
- postmortem Dec 24, 2025, 01:51 PM UTC
## Post-mortem: Partial Platform Outage ### Summary On December 22, 2025, between **10:17 and 13:40 UTC**, part of our CEP experienced service disruptions affecting message delivery, data ingestion, and dashboards for some customers. The incident was caused by a **hardware-related issue on a high-grade server** hosting part of our private virtualization infrastructure. The issue has since been fully resolved, and corrective actions have been implemented. ### Impact During the incident window, the following impacts were observed: * **Push and Email Campaigns \(CEP\):** Delivery was interrupted for a subset of campaigns. * **Data Ingestion & Analytics:** Temporary ingestion delays occurred. * **Dashboards:** The Batch dashboard was intermittently unavailable. No impact was observed on SMS delivery. The incident affected a limited subset of customers, depending on their usage at the time. ### Timeline \(UTC\) * **10:17** – First service degradation detected. * **10:20** – Investigation initiated. * **11:00** – Infrastructure issue identified on one hypervisor. * **11:30** – Mitigation and recovery actions started. * **13:40** – All impacted services fully operational. ### Root Cause The incident was caused by a **fault in the cooling system** of a high-grade server used in our private virtualization infrastructure. This cooling issue led to **overheating**, triggering a **protective shutdown of a disk group** on the affected server. As a result, the hypervisor abruptly lost access to multiple disks, causing all virtualized services hosted on that node to stop simultaneously. Although this class of hardware is designed to provide strong reliability guarantees, this cooling failure resulted in the loss of a single hypervisor and exposed the impact of service colocation on shared infrastructure. ### Resolution * The cooling issue and impacted hardware were fully repaired and validated by our infrastructure provider. * Affected services were restarted and resynchronized. * Data consistency was verified after recovery. ### Corrective and Preventive Actions Following this incident, we took the following actions: * The faulty hardware was repaired and removed from service until fully validated. * We reviewed service placement on our private virtualization infrastructure and **reduced the colocation of critical components** on single hypervisors. * Stateful services \(including Redis clusters\) were redistributed to **limit the blast radius** of a single host failure. * We **strengthened monitoring and alerting around service colocation**, allowing us to detect and act earlier when multiple critical components are unintentionally placed on the same underlying host. These actions aim to reduce the impact of similar infrastructure-level incidents in the future. ### Conclusion We apologize for the disruption this incident caused. While hardware failures of this nature are rare, this event highlighted areas where we could further improve infrastructure resilience and observability. We remain committed to transparency and continuous improvement. — **The Batch Engineering Team**
Notice November 22, 2025 CEP Delivery Issues 1h 55m
Started Nov 22, 2025, 09:30 AM UTC· resolved Nov 22, 2025, 11:25 AM UTC
- resolved Nov 22, 2025, 01:28 PM UTC
Between November 22nd at 10:30 CET and 12:08 CET, a performance issue on a database cluster impacted the delivery of Push notifications, SMS, and emails. Impact: * Event processing: Delayed by up to 1 hour and 30 minutes for a subset of users. * Campaign processing: Delayed by up to 2 hours for a subset of campaigns. * Delivery failures: A subset of campaign tasks timed out during processing and were not delivered. Resolution: A mitigation was applied at 12:08 CET, which restored normal processing performance. The resulting backlog was fully cleared by 12:25 CET.
Notice November 14, 2025 CEP Profile & Catalog API issues –
Started Nov 14, 2025, 09:30 AM UTC· resolved Nov 14, 2025, 09:30 AM UTC
- resolved Nov 14, 2025, 10:22 AM UTC
Between November 14th at 10:17 (CET) and November 14th at 10:31 (CET), the Profile API and Catalog API experienced an incident during which HTTP 500 errors were returned on all requests, impacting 100% of the traffic during this period. Those requests have not been processed and can be safely retried.
Minor October 30, 2025 Image display issues on older mobile devices 7h 10m
Started Oct 30, 2025, 09:30 AM UTC· resolved Oct 30, 2025, 04:40 PM UTC
- resolved Oct 30, 2025, 05:21 PM UTC
At 10:50 CET we performed a routine rotation of our TLS certificates; this unexpectedly caused images to appear blank in Push and In-App messages for some users. Affected devices: - iPhones/iPads running on iOS 17.7 or older - A very small percentage of Android devices The issue was resolved at 17:40 CET
Minor October 6, 2025 Profile updates and custom events processing delays on CEP 9h 12m Email delivery, Push Delivery, SMS Delivery, Data ingestion and 1 more
Started Oct 06, 2025, 11:45 AM UTC· resolved Oct 06, 2025, 08:57 PM UTC
- investigating Oct 06, 2025, 11:45 AM UTC
We are currently experiencing a delay in indexing messages from our SDKs. At the moment, message indexing is running approximately 1 hour behind. This delay impacts only the updating of Profile events and attributes. Specifically: • The profile view is not updated in real-time. • Some automations may be triggered later than expected.
- investigating Oct 06, 2025, 11:50 AM UTC
We are continuing to investigate this issue.
- monitoring Oct 06, 2025, 01:32 PM UTC
As of 15:15 (GMT+2), the situation is under control. Indexing delays on CEP have been resolved, Profile updates and custom event processing are now back to real-time. We’re continuing to monitor the platform to ensure stability.
- resolved Oct 06, 2025, 08:57 PM UTC
This incident has been resolved.
Minor October 6, 2025 Email composer maintenance 12d 23h
Started Oct 06, 2025, 08:28 AM UTC· resolved Oct 19, 2025, 08:00 AM UTC
- resolved Oct 19, 2025, 08:00 AM UTC
Oct 19, 08:00 UTC Completed - The scheduled maintenance has been completed. Oct 19, 07:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct 6, 08:28 UTC Scheduled - Our Email Composer partner will be performing an infrastructure update on October 19, 2025, from 09:00 to 10:00 CEST (UTC+2). During this window, the Email Composer may be unavailable. You won’t be able to: • Edit email messages in the composer • Edit your templates • Upload new images
Major August 22, 2025 CEP Automation triggers may stall in specific workflows 3d 21h Email delivery, Push Delivery, SMS Delivery
Started Aug 22, 2025, 09:11 AM UTC· resolved Aug 26, 2025, 06:56 AM UTC
- identified Aug 22, 2025, 09:11 AM UTC
Impact: In some cases, automation workflows that include Yes/No steps and/or Delay steps can stall. Affected profiles remain stuck and may miss subsequent communications. Estimated impact: ~1% of users. Scope: Limited to automations using both Yes/No and/or Delay steps. Other automations are not affected. Current status: We have identified the defect and are developing a fix. No workaround is available at this time. Next steps: The fix will be deployed after validation. We apologize for the disruption and appreciate your patience.
- identified Aug 22, 2025, 02:20 PM UTC
The fix is in testing and targeted for release early next week. Next update: Monday, August 25, by 18:00 CEST
- monitoring Aug 25, 2025, 07:48 AM UTC
A fix for this issue has been implemented, we are now monitoring the results.
- resolved Aug 26, 2025, 06:56 AM UTC
The issue affecting automation workflows with Yes/No and/or Delay steps has been resolved. The fix has been successfully deployed and results have been monitored without further impact. All CEP Core Services (Email, Push, SMS delivery) are now operating normally. We apologize for the disruption and thank you for your patience.
Minor August 14, 2025 CEP & MEP Delivery Issues 17h 14m Inbox, Email delivery, Push delivery, Push Delivery and 5 more
Started Aug 14, 2025, 12:23 PM UTC· resolved Aug 15, 2025, 05:37 AM UTC
- identified Aug 14, 2025, 12:23 PM UTC
We have been experiencing a database issue since August 14th, 12:10 CEST. We are working on a complete report of what is affected. In the meantime, here is a brief overview of the platform’s status * CEP Automations (Email, SMS, Push v2) are not being sent * MEP Automations (Push v1) are not being sent * MEP Campaigns are partially being sent * CEP Campaigns are partially being sent * The transactional API (MEP) is working * Data sent on APIs for which we sent back a 200 HTTP status code may not be processed live but is enqueued and will eventually be processed * Some APIs might experience elevated error rate over the afternoon * Webhooks are partially working We will provide further updates as soon as we have more information or in 1 hour, whichever comes first.
- identified Aug 14, 2025, 01:33 PM UTC
We are still working on fixing the database problem. The affected components are still the same.
- identified Aug 14, 2025, 02:38 PM UTC
The database issue has been resolved. All components are gradually returning to normal. Processing speed is being increased progressively, with priority given to maintaining system stability. We will provide further updates as soon as we have more information or in 1 hour, whichever comes first.
- identified Aug 14, 2025, 03:36 PM UTC
The system is still catching up with the backlogged work. All late CEP & MEP Automation messages (Email, SMS, Push v1 & v2) will be sent. We will provide further updates as soon as we have more information or in 1 hour, whichever comes first.
- monitoring Aug 14, 2025, 04:04 PM UTC
The system has now fully processed the backlog of CEP and MEP automation messages, including Email, SMS, and Push v1 and v2. All previously delayed messages have been sent. We are now moving into a monitoring phase, during which the on-call team will closely watch system performance.
- resolved Aug 15, 2025, 05:37 AM UTC
The system has been stable for the last 12 hours. As of August 14, 18:04 CEST: - Late MEP & CEP Trigger Automation messages have been sent - Some MEP Push Campaigns & Recurring Automations have been partially send and will not be retried - API error rates are back to normal - Webhooks are back to normal This incident is now considered as resolved.
- postmortem Aug 21, 2025, 07:07 AM UTC
On August 14th, 2025, we encountered a database issue when performing a maintenance operation. This issue resulted in a major outage of our MEP & CEP delivery services. ## What happened? During the rework of our internal DNS, we performed the renaming of servers using our IaC \(Infrastructure as Code\). A quirk of one of our IaC plugins can cause it to wrongly believe that machines should be re-imaged. We knew about this issue and applied a safeguard, but this safeguard wasn’t enabled on 5 of the database instances. When applying the IaC changes, those 5 servers were wiped clean of their data and re-imaged. The erased nodes caused some data to be unavailable \(~0.028% of total data\) as all of their replicas were on the deleted machines. It also broke cluster quorum, forcing services to operate at a degraded consistency level, which raised their error rates. Core services used by both the MEP & CEP were on this database cluster. In some cases, services stopped processing events when they encountered unreadable database rows to ensure data integrity. The issue was immediately noticed, and our incident response system was triggered. It was fixed by restoring a backup of the wiped nodes. We then manually brought the system back online by slowly restoring services so that the database wasn’t overloaded while the applications caught up with the backlogged work. ## Impact on our platform * MEP: Push v1 Campaigns & Trigger Automations were unavailable during the incident. Trigger automations were eventually sent, but many campaign messages were dropped. Campaigns sent between August 13th and 14th may be missing analytics. * CEP: Automations \(Push v2, SMS, Email\) were backlogged during the duration of the incident. * APIs: Experienced elevated error rates, especially around 12:10 CEST when the incident started. # **Timeline & mitigation actions** _For clarity, this timeline only lists the most important events. All times are CEST._ **12:10** The IaC operation was applied, and the 5 servers were re-imaged. **12:20** Our infrastructure engineers were paged about the missing database nodes and started looking at the problem. The incident response process was triggered, calling other engineers to help. They began drafting an action plan. **12:40** The action plan was reviewed, and work on it started. Database backups were being restored. A second plan was drafted, hoping that it could help us bring part of the system online sooner than expected. We started an in-depth audit of all of our products to establish a list of what was working and what was not **13:30–14:23** The severity of the incident was reevaluated; the Statuspage incident was drafted and published. **15:50** The database restore was finished, and the cluster was fully back online. As we expected, human intervention was required to carefully bring the services back online while managing database load as the system needed to catch up with the backlogged work, which we immediately started doing. **15:30–17:58** The system caught up with the backlogged work, and the database cluster was stable. We wrote an update and moved the incident to the “Monitoring” phase. **07:37, the next morning** The on-call team reported that everything had been working as expected since we moved to “Monitoring”. We marked the incident as resolved. ## **Forthcoming actions** * We are looking at ways to split this database cluster into smaller ones to reduce the blast radius of an unavailability event. * We improved our IaC configuration to add more protections against accidental re-imaging. * We will be making changes to how the IaC action plan is presented to human operators by reducing noise to make it less likely to miss an unwanted change.
Minor August 7, 2025 Delays in CEP automation triggers (Push, Email & SMS) 1h 4m Email delivery, Push Delivery, SMS Delivery
Started Aug 07, 2025, 09:21 AM UTC· resolved Aug 07, 2025, 10:25 AM UTC
- identified Aug 07, 2025, 09:21 AM UTC
Following a maintenance operation on our data‑storage system, CEP automation triggers have been processing more slowly than expected. Our engineering team is analysing the issue and working to restore normal performance. We will provide further updates as soon as we have more information.
- monitoring Aug 07, 2025, 10:12 AM UTC
We have rolled back the database maintenance operation, and all automation triggers have now processed their backlog. All CEP campaigns with Push, Email & SMS are now back online. We are actively monitoring the situation and will provide further updates as we progress.
- resolved Aug 07, 2025, 10:25 AM UTC
This incident has been resolved.
Notice August 6, 2025 Open Push Event Indexing Issue on CEP Platform 42m
Started Aug 06, 2025, 03:53 PM UTC· resolved Aug 06, 2025, 04:35 PM UTC
- monitoring Aug 06, 2025, 03:53 PM UTC
Since May 15, around 16:00 CEST, we have been experiencing issues indexing “open push” events on the CEP platform, impacting all push types. This indexing issue led to partial data loss, causing incomplete or missing data in dashboards, APIs, exports, and affecting retargeting campaigns based on open events. Not all data was impacted, but some events remain unrecoverable. A fix has been deployed, and you may now observe an increase in open rates compared to previous days. We will keep you informed of any further updates on this page.
- resolved Aug 06, 2025, 04:35 PM UTC
This incident has been resolved.
Major July 31, 2025 Delay on CEP Trigger Automations 38m Email delivery, Push Delivery, SMS Delivery
Started Jul 31, 2025, 08:27 AM UTC· resolved Jul 31, 2025, 09:05 AM UTC
- investigating Jul 31, 2025, 08:56 AM UTC
Starting July 30 at 00:13 CEST, we began experiencing intermittent delays affecting CEP Trigger Automations. During the night, the issue was limited to email channels and impacted approximately 2% of traffic. The problem escalated between 10:20 and 10:44 CEST on July 31, affecting all triggered sends across email, SMS, and push channels, with delays of approximately 15 minutes. The backlog has since been cleared. We are currently implementing a permanent fix and continuing to monitor the system closely. No further delays have been observed.
- resolved Jul 31, 2025, 09:05 AM UTC
A fix has been deployed to production, and CEP Trigger Automations are now operating normally across all channels. Additional tests have been added to prevent this issue from recurring.
Minor July 24, 2025 Indexing delay affecting Profile autocomplete and analytics –
Started Jul 24, 2025, 08:30 PM UTC· resolved Jul 24, 2025, 08:30 PM UTC
- resolved Jul 25, 2025, 07:39 AM UTC
Since 24 July at 22:34 CEST, we are experiencing an indexing delay in part of our indexing pipeline. This issue affects: - The availability of new Profile attributes in the dashboard autocomplete - Profile analytics, which may display inaccurate data We are actively working to resolve the issue and will provide updates as soon as possible.
Notice July 22, 2025 Temporary Service Disruptions Following Network Maintenance 34m
Started Jul 22, 2025, 01:30 PM UTC· resolved Jul 22, 2025, 02:04 PM UTC
- resolved Jul 22, 2025, 04:05 PM UTC
On July 22 from 15:34 to 16:04 CEST, the Dashboard and some APIs experienced temporary disruptions due to unexpected effects during a planned network maintenance. Impacted components included the Dashboard, Data ingestion, Trigger Events API, Transactionnal API, Profile API, and Custom Audience API. The issue has been resolved and all systems are now operating normally.
Minor July 13, 2025 Delays in CEP Trigger Automations 3h 8m
Started Jul 13, 2025, 04:00 PM UTC· resolved Jul 13, 2025, 07:08 PM UTC
- resolved Jul 13, 2025, 05:33 PM UTC
Between 18:05 and 19:08 GMT+2, our Trigger Automations component experienced an issue that prevented new trigger events from being processed in real-time. This resulted in delayed communications, including emails, push notifications, and SMS, for up to one hour. Our engineering team identified and resolved the underlying issue, restoring normal operations by 19:06 GMT+2. A backlog of delayed events was fully processed by 19:08 GMT+2. All systems are now operating normally. We sincerely apologize for any disruption this may have caused.
Critical July 7, 2025 Dashboard unreachable 15m Dashboard, Dashboard, [CEP] Audience API
Started Jul 07, 2025, 09:58 AM UTC· resolved Jul 07, 2025, 10:14 AM UTC
- investigating Jul 07, 2025, 09:58 AM UTC
We are currently investigating this issue.
- investigating Jul 07, 2025, 09:59 AM UTC
We are continuing to investigate this issue.
- investigating Jul 07, 2025, 10:06 AM UTC
We are currently investigating a network issue that is affecting access to the main dashboard. Our engineering team is actively working to identify the root cause and restore normal service as quickly as possible.
- investigating Jul 07, 2025, 10:06 AM UTC
We are continuing to investigate this issue.
- investigating Jul 07, 2025, 10:09 AM UTC
Dashboard is back online, we are still investigating.
- resolved Jul 07, 2025, 10:14 AM UTC
This incident has been resolved.
Minor July 3, 2025 Incorrect Date Attribute Values on Some CEP Profiles 17h 34m
Started Jul 03, 2025, 05:04 PM UTC· resolved Jul 04, 2025, 10:39 AM UTC
- identified Jul 03, 2025, 05:04 PM UTC
We have identified an issue affecting Date attributes on Profiles when updated via our SDKs (v2.x and v3.x). Between July 2nd, 15:36 CET and July 3rd, 15:25 CET, any Date attribute set using the SDK was assigned an incorrect value far in the future. Impact As a result of this issue: * Targeting based on these attributes will not behave as expected. * Profiles will not appear correctly in the Dashboard’s Profile View. * Data exports will reflect incorrect dates for affected attributes. What is Not Affected * Attributes updated outside the impacted time window have the correct value. * Attributes updated via the Profile API or Custom Data API are not impacted. * The MEP (Mobile Engagement Platform) remains unaffected. Current status Our engineering team is actively working on restoring the correct values for all impacted Profiles. As it is a lengthy process, we expect this work to be completed in the next 24h.
- resolved Jul 04, 2025, 10:39 AM UTC
The data has been restored: * Targetings using those attributes will now work as expected * The attributes now show up in the Profile View * Future exports will show the correct data We will not replay automations & campaigns to profiles that should have been targeted but were not. Future occurrences will work as expected.
Major July 1, 2025 Network Issue Impacting Message Delivery and APIs 4h 57m Dashboard, Editorial dashboard, Dashboard, [CEP] Profile API and 31 more
Started Jul 01, 2025, 02:42 PM UTC· resolved Jul 01, 2025, 07:40 PM UTC
- investigating Jul 01, 2025, 02:42 PM UTC
Since 15:54 CET, we have been experiencing a network-related incident with our hosting provider. As a result: - A majority of outgoing messages (push notifications, emails, and SMS) are not being sent, across both our MEP and CEP platforms. - All APIs are impacted and are currently showing an elevated error rate. - The dashboard remains accessible, but its performance is significantly degraded. Our engineering team is actively investigating the root cause and working to restore full service as quickly as possible. We will provide an update by 17:36 CET or as soon as new information becomes available.
- identified Jul 01, 2025, 03:27 PM UTC
As of now, the incident is still ongoing. We are currently waiting for our hosting provider to fully resolve the underlying network issue. Impacted servers are progressively coming back online, which is a positive indication toward recovery. Our team continues to closely monitor the situation and remains ready to take immediate action as soon as full service restoration becomes possible. We will share another update by 18:36 CET, or sooner if we have meaningful progress to report.
- monitoring Jul 01, 2025, 04:00 PM UTC
As of 17:50 CET, all previously impacted servers have been brought back online. All services are now operational. However, some functionalities may still experience degraded performance while we continue to actively monitor the platform. Our engineering team remains fully mobilized to ensure complete recovery and platform stability. We will provide a final update once the incident is officially resolved.
- resolved Jul 01, 2025, 07:40 PM UTC
As of 21:39 CET, the incident has been fully resolved. All services are back to normal, with no remaining latency or performance degradation. The platform is stable and operating as expected. A detailed post-mortem will be published shortly to provide transparency on the root cause and the remediation actions taken. We thank you for your continued trust and patience throughout this incident.
- postmortem Jul 29, 2025, 08:52 AM UTC
### What happened On July 1, starting at 15:54 CET, a network malfunction disrupted connectivity for a portion of our bare‑metal infrastructure. As a result, message delivery \(email, push, SMS\) and several APIs suffered partial outage or degraded performance. The disruption persisted for roughly 2 hours, with full restoration by approximately 18:00 CET. ### Why it happened The incident was caused by two redundant network switches that failed to recover correctly after a power issue. Both switches had a firmware defect that prevented synchronization, leading to network partition across affected racks. The manufacturer has since identified and fixed the firmware bug, which is no longer in operation. ### What we’ve done In coordination with our hosting provider, we restored network connectivity and progressively brought all impacted systems back online. By approximately 20:54 CET, service levels had fully normalized. No customer data has been lost during the incident, though data ingested through our public APIs and SDKs during the window may not have been processed if not retried. ### What we’re doing next To improve resilience, we are enhancing our infrastructure monitoring to detect hardware-level network issues faster and with greater precision. We’re also continuously improving our recovery tools and incident response processes to streamline restoration during such events. We sincerely apologize for the disruption and appreciate the trust you place in us. If you have any questions or would like further clarification, please feel free to reach out.
Major June 26, 2025 Image loading issues 2d 1h Dashboard, Editorial dashboard, Dashboard, Push delivery and 1 more
Started Jun 26, 2025, 08:31 AM UTC· resolved Jun 28, 2025, 09:52 AM UTC
- identified Jun 26, 2025, 08:31 AM UTC
We're currently experiencing intermittent errors with our image loading system. The impact is two fold: * some campaigns cannot be created correctly due to image loading failures, affecting both the main dashboard and the editorial dashboards. * some transactional push notifications containing images cannot be sent correctly due to image loading failures. This issue started on June 20th at 17:00 CET. Our team is actively investigating the issue and working to restore full functionality as quickly as possible. We apologize for any inconvenience this may cause.
- identified Jun 26, 2025, 04:26 PM UTC
We're still observing intermittent image loading errors, though at a significantly reduced rate compared to the last 24 hours. The last reported error occurred at 15:00 CET. Our team continues to actively work on implementing a permanent fix.
- monitoring Jun 27, 2025, 10:18 AM UTC
A fix has been implemented. We haven't observed any new issues for the past two hours and will continue to monitor the situation closely.
- resolved Jun 28, 2025, 09:52 AM UTC
This incident has been resolved.