Blackthorn experienced a critical incident on September 26, 2024 affecting Blackthorn Messaging, lasting 1h 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 26, 2024, 04:59 PM UTC
Our Engineering team is currently investigating an ongoing issue impacting SMS deliverability. We will provide an update shortly.
- resolved Sep 26, 2024, 06:16 PM UTC
The SMS deliverability issue has been successfully identified and resolved. We will provide a detailed Root Cause Analysis for this incident and apologize for this system outage with the Messaging application.
- postmortem Oct 01, 2024, 09:53 PM UTC
Our engineering team has identified the root cause of the SMS Application outage. We experienced a brief service interruption on September 26, 2024, between 12:00 PM and 2:00 PM EDT, which impacted all of our messaging services. Our internal load-balancer auto-scaling was stuck, which required our Engineering team to modify the web server. During this time, customers were unable to send or receive messages, and updates related to Stripe failed as well. Services were fully restored within two hours, and we ensured that no data was lost by verifying all processes through our health-check systems and API. Going forward, our Engineering team will generate additional Fallback DNS configuration to pass requests to another resource for processing redundancy to avoid further disruption.