Braze incident

SparkPost email delivery latency in EU region

Minor Resolved View vendor source →

Braze experienced a minor incident on November 29, 2024 affecting Email, lasting 3h 27m. The incident has been resolved; the full update timeline is below.

Started
Nov 29, 2024, 09:50 AM UTC
Resolved
Nov 29, 2024, 01:17 PM UTC
Duration
3h 27m
Detected by Pingoru
Nov 29, 2024, 09:50 AM UTC

Affected components

Email

Update timeline

  1. identified Nov 29, 2024, 09:50 AM UTC

    SparkPost, one of the vendors Braze uses for email delivery, is experiencing an incident in the EU region. As a result, Braze is currently observing an increase in message-sending latency for emails via SparkPost. All Braze’s systems are within normal operating bounds. We have escalated the issue to senior members of the SparkPost support team and are awaiting further information. SparkPost has confirmed via its status page (https://status.sparkpost.com/incidents/t987cly748ll) that it is “seeing improvements in message processing. No messages are lost.”

  2. monitoring Nov 29, 2024, 10:20 AM UTC

    SparkPost has confirmed that a fix has been implemented. The email-sending latency is decreasing, new messages are being accepted and sent, and no messages have been lost. We continue to monitor the situation and liaise with SparkPost.

  3. monitoring Nov 29, 2024, 11:11 AM UTC

    SparkPost has confirmed that a solution has been implemented, and they expect the message backlog to be processed by 11:20 UTC. New messages are being accepted and delivered without delay.

  4. monitoring Nov 29, 2024, 11:48 AM UTC

    We continue to monitor the situation and are awaiting further updates from SparkPost.

  5. monitoring Nov 29, 2024, 12:48 PM UTC

    SparkPost has shared an update on their Status Page to inform that the email queues are drained, except for a single Braze IP Pool. We will continue to work with SparkPost and provide an update as soon as this incident is fully resolved.

  6. resolved Nov 29, 2024, 01:17 PM UTC

    SparkPost has completed the processing of the backlog for the majority of Braze IP Pools. One IP Pool remains impacted and Braze is working directly with the impacted customer. We will work with SparkPost to obtain root cause details of this issue. We will now mark the StatusPage as resolved.