Braze incident
Currents and CDI latency impacting EU-01 and EU-02
Braze experienced a major incident on March 10, 2025 affecting Outbound Messaging and Outbound Messaging and 1 more component, lasting 6h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 10, 2025, 04:27 AM UTC
We are currently experiencing latency issues with Currents and CDI due to an infrastructure problem. Our engineering team is actively working to resolve this and restore normal functionality as quickly as possible.
- identified Mar 10, 2025, 05:06 AM UTC
We continue to observe elevated latency impacting Currents and CDI. Our engineering teams are actively working to resolve this and restore normal functionality as quickly as possible.
- identified Mar 10, 2025, 06:17 AM UTC
We continue to observe elevated latency impacting Currents and CDI. Our engineering teams are actively working to resolve this and restore normal functionality as quickly as possible.
- identified Mar 10, 2025, 07:11 AM UTC
Engineers continue to work to resolve the issue impacting Currents and CDI. Latency is currently highly elevated for Currents. Additionally, some customers may experience latency with CDI integrations.
- identified Mar 10, 2025, 08:16 AM UTC
Braze Engineers are actively working to resolve the ongoing issue impacting Currents and CDI. We are now observing elevated messaging latency as a result of this issue. Currently, latency remains highly elevated for Currents, and some customers may also experience latency with CDI integrations. We appreciate your patience as we work to resolve these issues.
- identified Mar 10, 2025, 11:15 AM UTC
The Braze Engineers have deployed changes to decrease Messaging latency and most messages are sent within normal operational bounds. Our Engineers continue to work on restoring the Currents and CDI Services. At this time Currents and CDI Services latency remains very high.
- identified Mar 10, 2025, 12:56 PM UTC
Outbound Messaging is operating within normal bounds following the changes deployed by The Braze Engineers. At this time Currents and CDI Services latency remains very high and our Engineers are continuing to work to restore the Currents and CDI Services.
- identified Mar 10, 2025, 03:47 PM UTC
Outbound Messaging is continuing to operate within normal bounds. The Braze CDI is now fully operational, and all new events are being processed within normal operational bounds. If you do not regularly sync the CDI integration, you will need to replay any syncs that failed to fully complete. This step is necessary to prevent old data from overwriting new data that was successfully synced during the incident. You can check the status of your jobs in the dashboard under the Sync log tab within the Cloud Data Ingestion page. Our Braze Engineers are actively working to restore the Currents service. We will provide further updates as the restoration progresses.
- identified Mar 10, 2025, 04:35 PM UTC
Outbound messaging continues to operate within normal bounds. The Braze CDI continues to be fully operational, and all new events are being processed within normal operational bounds. The Currents Service has been restored, and new events are being processed within normal operational bounds. Braze Engineers continue to work on processing the backlog of events impacted by the incident.
- monitoring Mar 10, 2025, 06:48 PM UTC
Currents is functioning normally, with all new events processing within normal operational bounds. Our Braze Engineering team is actively working to clear the backlog of Currents events.
- monitoring Mar 10, 2025, 07:01 PM UTC
We are continuing to monitor for any further issues.
- resolved Mar 10, 2025, 10:21 PM UTC
As of 11:20am ET, both the Currents and Braze CDI services have been fully restored to normal operational performance. Since that time, all CDI syncs and new Currents events have been processing within standard parameters. The Braze Engineering team has completed the immediate restoration work and is continuing to process the backlog of Currents events produced during the incident's impact window. Depending on the volume of events and the downstream destination for your integration, the backlog processing may take some time and will be different for each integration. As such, with systems operating normally for new incoming events, we are marking this issue as resolved at this time. Should you have questions, please reach out to Braze Support.