Is Batch down?

We are investigating an issue affecting the Batch dashboard and message delivery. Since around 12:20 CET, the dashboard may be slow, fail to load campaign data or log users out, and most push notifications and emails (campaigns, automations and transactional) are not being sent. Transactional API calls are still accepted, but the related messages are currently not delivered. Our team is actively working on a resolution.

We restored the affected internal service at 12:58 CET, and the dashboard is back to normal. Message delivery has resumed: push notifications and emails delayed since 12:20 CET are currently being sent, so some messages may still arrive late. We're still searching for the root issue. We are monitoring until delivery is fully back to real time.

On November 2, between 12:19 and 12:58 CET, an internal database connectivity issue caused the dashboard to be very slow and delayed message sending (push, email and transactional API). The issue was mitigated at 12:58 CET: the dashboard is back to normal and the sending backlog has been processed. Campaigns and automations scheduled during this window were delayed and have since been sent. We are continuing to monitor the situation.

This incident has been resolved. On November 2, between 12:19 and 13:02 CET, a brief network interruption between our database servers left one of our internal services unable to reconnect, which made the dashboard and the Campaigns API very slow or unavailable and delayed message sending (push, email, SMS and transactional). Service was fully restored at 13:02 CET, and all delayed messages were processed by 13:10 CET. We are adding safeguards so that database connections recover automatically from this kind of interruption.

On October 26, between 09:54 and 11:30 CET, a subset of Android push notifications sent via the Transactional API experienced intermittent delays of up to 15 minutes, and some were delivered multiple times to the same device. iOS push notifications, campaigns and automations were not affected. This was caused by a bug on our side, triggered by an edge case during scheduled maintenance performed by our infrastructure provider. Deliveries have been back to normal since 11:30 CET. The bug is still present and the issue could happen again: we are monitoring closely and working on a fix.

This incident has been resolved.

A security vulnerability affecting widely used open-source web software (PHP) was publicly disclosed today. This is not an incident on Batch's side: we have no indication that our platform or our customers' data have been affected. We advise that you check your own systems to see if they are affected by this issue. As a precaution, we are temporarily disabling file uploads in the Batch dashboard while we wait for the necessary updates to be released and apply them. During this time, at least the following actions are unavailable: - Creating audiences from a file - Uploading or updating iOS and Safari push certificates and keys - Updating a landing page favicon Everything else keeps working normally: SDKs, push sending, campaigns, automations, and APIs are not affected. We expect to restore these features within a few hours and will post an update here as soon as they are back.

Affected Batch systems have been fully upgraded to patched versions.

We have identified an issue affecting our profile processing pipeline since 15:46 CEST. Profile updates, including attributes, events and delivery statuses, are currently delayed by approximately 15 minutes and the delay is growing. Automations triggered by these updates may also be delayed. A fix has been identified and is being deployed. No data has been lost, all updates will be processed as the backlog clears.

A fix was deployed at 16:08 CEST and profile processing has resumed. Updates sent through the Profile API are processed in real time again. Updates sent by our mobile SDKs, such as attributes and events, are still delayed while the backlog clears, and automations triggered by these events may also be delayed. We are monitoring the recovery.

This incident has been resolved. Profile updates and the automations triggered by them were delayed between 15:46 CEST and 16:35 CEST. The fix deployed at 16:08 CEST has restored normal processing, and all delayed updates have now been processed. All components are operational.

We have identified an issue affecting audience imports from files delivered via FTP/SFTP. Since October 12, these files have not been processed, so the related audiences have not been updated. A fix is being deployed and imports are resuming. A backfill of the files received during this period will follow.

Audience imports from files delivered via FTP/SFTP have resumed. Files received since October 12 are being processed automatically, and the related audiences are being updated as the backlog clears. We are monitoring the import pipeline until it is back to normal.

This incident has been resolved. Audience imports from files delivered via FTP/SFTP are back to normal, and files received during the incident have been processed.