Blackthorn experienced a major incident on November 10, 2021 affecting Blackthorn Events, lasting 3h 35m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 10, 2021, 04:50 PM UTC
We’ve been experiencing intermittent errors with AttendeeLinks and Donation pages this morning and our developers are closely monitoring app traffic. Just like all other Salesforce end users, we are also subject to Salesforce API limits, which govern available resources to ensure stable infrastructure performance for all. Unusually high traffic coupled with the known limitations of the Salesforce platform cause the intermittent 400 error. While we can’t change the governing limits of Salesforce, we have some infrastructure changes to our apps being prioritized over the next few releases to compensate for those Salesforce limits and improve performance.
- monitoring Nov 10, 2021, 06:43 PM UTC
A fix has been implemented and we are monitoring the results. Events and Donation pages appear to be loading normally. We are doing further investigation into the root cause of the incident and will follow up with more information once this has been completed. Please contact support at https://community.blackthorn.io/s/support if you have any questions.
- resolved Nov 10, 2021, 08:26 PM UTC
This incident has been resolved. The root cause was a combination of Salesforce API limitations, unusally high traffic events running concurrently, and a limitation in the amount of connections allowed in one of our databases. In the short term we have doubled our database capacity to ensure we no longer hit any connection limits. In the long term we have established a plan of refactoring our infrastructure to reduce the impact of the Salesforce API limitations. Please contact support at https://community.blackthorn.io/s/support if you have any questions.