Blackthorn experienced a major incident on March 15, 2023 affecting Blackthorn Events, lasting 14h 47m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 15, 2023, 10:46 PM UTC
Our Engineering teams are currently investigating an issue with Event URLs not updating after changes are made to an Event record in Salesforce. More information about this issue can be found here: https://community.blackthorn.io/s/article/ET-1951 As our teams continue their investigation, a temporary solution to resolve this issue is to initiate a hard refresh of the impacted Event URL.
- resolved Mar 16, 2023, 01:34 PM UTC
Our Engineering team has resolved the Event URL display issue and determined the root cause. We will continue to actively monitor the situation. This incident was related to third-party hosting infrastructure that experienced an issue with insufficient resources. This issue is now resolved and customers should see immediate updates to their Event URLs when changes are made. We will increase our monitoring of those services going forward.
- postmortem Mar 16, 2023, 06:56 PM UTC
Hello Blackthorn Customers - We would like to apologize for the caching issue, which was resolved this morning. We understand for many of you this may have had a significant business impact if you were attempting to update in-flight events with heavy traffic, or needed to delay a new event launch. We believe this began at the beginning of yesterday \(the inability to update a live event\). Events did not go down, just the ability to update them wasn’t available. After further review of the root cause with our engineering team, we wanted to send over a clarification to the email sent earlier today. Our platform was allocating insufficient resources to our caching infrastructure \(on AWS\). The monitoring of this particular system resource was not specific enough to pick up on this overflow. Lastly we did not size the resource for caching properly and at the time, did not have a notification system in place to indicate that traffic was not processing even though the workers \(system processes\) were live. To address this moving forward, we’ve increased the available resources and are enhancing our ability to more granularly monitor production resources. If you run into any issues or questions going forward, please reach out to our support team here: [https://community.blackthorn.io/s/support](https://community.blackthorn.io/s/support) Sincerely, Blackthorn Team