Blackthorn experienced a critical incident on June 19, 2023 affecting Blackthorn Messaging, lasting 21h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 19, 2023, 06:22 PM UTC
We are currently experiencing an outage with Blackthorn Messaging. We are working to restore service.
- monitoring Jun 19, 2023, 07:28 PM UTC
Blackthorn Messaging App service has been restored. We are continuing to monitor the situation and will provide a Root Cause Analysis regarding this issue.
- resolved Jun 20, 2023, 03:54 PM UTC
The Messaging Application performance issue has been confirmed resolved by our Engineering team. We will provide a Root Cause Analysis regarding this incident.
- postmortem Jun 21, 2023, 03:37 PM UTC
On June 19, 2023, following a scheduled system maintenance, our messaging services experienced a significant interruption. The cause was linked to a recent update in our system infrastructure, which didn't perform as expected. This issue affected all messaging operations, including sending, receiving, and scheduled deliveries for 2 hours and 11 minutes. As soon as the problem was detected, our response team promptly restored the services by redirecting endpoints to system resources part of our infrastructure. We are working to improve our system's monitoring to better detect issues like this in the future. To reduce the chance of a similar incident, we have already initiated automated system health checks for early detection of similar instance.