Blackthorn experienced a major incident on March 31, 2023 affecting Blackthorn Events, lasting 1h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 31, 2023, 10:33 PM UTC
Our Engineering teams are currently investigating a caching issue with Event URLs not updating after modifications are made to an Event record in Salesforce. More information about this issue can be found here: https://community.blackthorn.io/s/article/ET-1998 As our teams continue their investigation, a temporary solution to resolve this issue is to initiate a hard refresh of the impacted Event URL.
- resolved Mar 31, 2023, 11:58 PM UTC
Our Engineering teams have resolved this issue and we are continuing to monitor the situation. We will provide a Root Cause Analysis of this issue on Monday, April 3rd.
- postmortem Apr 03, 2023, 07:52 PM UTC
Hello Blackthorn Customers - We would like to apologize for the recent caching issue, which was identified and resolved last Friday. We understand for many of you this may have had a significant business impact if you were attempting to update in-flight events with heavy traffic, or needed to delay a new event launch. Our platform experienced degraded performance due to a spike in cache invalidation requests. We believe this issue of an inability to update events began on Friday afternoon and was resolved after approximately two hours and thirty-five minutes. Event pages or registrations did not go down, only the ability to update Event pages. To address this moving forward, we will update our incident resolution steps and work to improve our monitoring and auto-scaling automation. If you run into any issues or questions going forward, please reach out to our support team here: [https://community.blackthorn.io/s/support](https://community.blackthorn.io/s/support) Sincerely, The Blackthorn Team