Blackthorn incident
Blackthorn Messaging - Outage Impacting Ability to Send Messages
Blackthorn experienced a critical incident on March 11, 2024 affecting Blackthorn Messaging, lasting 1h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 11, 2024, 08:05 PM UTC
Our Engineering team is currently investigating an issue relating to the inability to send messages. We will provide updates here as the investigation continues. Thank you for your patience as we urgently work to restore SMS deliverability.
- investigating Mar 11, 2024, 09:35 PM UTC
Messaging App service is being restored, and we will confirm when this is complete. We are continuing to monitor the situation and will also provide a Root Cause Analysis regarding this issue.
- investigating Mar 11, 2024, 09:51 PM UTC
Our Blackthorn Messaging App service has now been restored. We are currently monitoring the situation but please reach out to Blackthorn Support at https://community.blackthorn.io/s/support-request if you run into any issues going forward.
- monitoring Mar 11, 2024, 09:54 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 11, 2024, 09:54 PM UTC
This incident has been resolved.
- postmortem Mar 11, 2024, 11:01 PM UTC
After further investigation, our engineering team has identified the root cause. It appears the Database supporting production services reached it's limit preventing customers from being able to send messages. This service had to be expanded to restore messaging, and should continue working as expected. Moving forward, Autoscaling and monitoring tools will be enabled as part of the resolution so this shouldn’t happen again.