Is Braze down?

We are currently investigating degraded performance in our EU02 cluster, which is causing delays in Currents processing for a small subset of customers. No data has been lost, and the majority of Currents data continues to flow normally. Our team is actively working toward a resolution and will provide updates as they become available.

We continue to investigate degraded performance in EU02. We have identified the root cause of the underlying latency, which stemmed from multiple Currents connectors failing to gracefully restart after routine maintenance, but appearing in a healthy state in our monitoring system. Those specific servers build significant backlogs while in this state. We have completed a graceful restart, and have spun up additional resources to process through the backlogged items more rapidly. At this point ~95% of data is flowing normally, while up to ~5% of our connectors, impacting a subset of customers, continue to experience latency. We see throughput significantly improved due to config changes, and will provide an update in 1 hour or when we have an updated timeline.

We are continuing to work on a fix for this issue.

This incident is resolved. As of 15:18 ET / 19:18 UTC, we have fully processed the backlog of jobs, and our EU02 Currents connectors are all again processing within normal latency ranges. No data was lost, and all Currents events have been sent to their destinations.

At approximately 7:30 AM ET, we became aware of an AWS networking outage in the US-WEST-2 region affecting network connectivity between 6:55 AM ET and 8:00 AM ET. AWS has since resolved the underlying issue. During the impact window, some customers may have experienced intermittent delays and latency in email delivery. Mail is now being delivered normally. As queue depth increased, some customers may have experienced residual delays in message delivery. These delays have been resolved as of 2:40 PM ET today. This disruption originated within AWS infrastructure and was not caused by any component of the Braze platform. For further details on the AWS incident, please refer to the AWS status page: https://health.aws.amazon.com/health/status

Braze engineers have identified an issue affecting the Braze AI Operator, where users may see the following error message: "Connection unstable - Check your Wi-Fi or internet signal and try again". We are actively investigating the issue and will provide updates as more information becomes available.

A fix is being deployed for the issue affecting the Braze AI Operator, where users were seeing the error message: "Connection unstable - Check your Wi-Fi or internet signal and try again." We are seeing error rates improve and are continuing to monitor closely to confirm full resolution. We will provide a further update once we can confirm the issue is fully resolved.

The issue impacting the Braze AI Operator, where users were seeing the error message: "Connection unstable - Check your Wi-Fi or internet signal and try again," has been fully resolved. The fix has been deployed and error rates have returned to normal levels.

Meta, a third-party, is currently experiencing an issue impacting WhatsApp services. During this time, WhatsApp messages may experience latency. Campaign and Canvas creation or modification may experience errors and latency when utilizing WhatsApp templates. Audience sync to Facebook may also experience errors and latency during this period.

Meta's services appear to be stabilizing. We are continuing to monitor the situation and will provide a further update once full service has been restored.

Meta has recovered from the earlier issue impacting WhatsApp services, and full service has now been restored. WhatsApp message delivery, Campaign and Canvas creation or modification using WhatsApp templates, and Audience sync to Facebook are all operating normally.

Starting at 1:00 PM ET, customers on our US-01 and EU-01 clusters are experiencing elevated latency affecting campaign and Canvas processing, data ingestion, and message sending. Our engineering team is actively investigating, and we will provide updates as the situation develops.

Our engineering team continues to investigate the elevated latency. We will provide another update in 30 min or sooner if there are any changes.

Our engineering team has implemented a change, and we are beginning to see improvements in latency across campaign and Canvas processing, data ingestion, and message sending on US-01 and EU-01. We are monitoring the situation closely.

Latency and error rates have remained at normal levels across all services following the implementation of our fix. This incident is resolved.