Blackthorn experienced a critical incident on February 1, 2022 affecting Blackthorn Events, lasting 3h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 01, 2022, 12:57 AM UTC
Many Events and AttendeeLink hosted pages are current displaying "Error (4xx)". Our team is actively investigating this issue and we will provide regular updates as more information becomes available.
- monitoring Feb 01, 2022, 01:25 AM UTC
All Events and AttendeeLink pages now appear to be loading properly. We are investigating a temporary, large spike of requests responsible for this short outage. We are currently monitoring the situation closely and we will be conducting a root cause analysis to fully understand this issue.
- resolved Feb 01, 2022, 04:08 AM UTC
This incident has been resolved.
- postmortem Feb 01, 2022, 02:58 PM UTC
On Monday 2/1, Blackthorn Events had some downtime beginning around 7:30pm EST was resolved approximately 90 minutes later. This was due to a large spike of requests responsible for this short outage impacting Blackthorn Event Pages. Please note Events and AttendeeLink pages are now loading properly. Please find additional details below: * Root Cause: Server crashed due to extensive amount of resources being consumed due to DNS errors * Solution: * Increasing our Heroku limits to prevent future timeouts * Move from Connect 360 to AWS so we can scale more efficiently