Thought Industries experienced a major incident on October 21, 2024 affecting US - Platform, lasting 1h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 21, 2024, 02:26 PM UTC
We are investigating high loading times on Rustici SCORM launches in the US.
- investigating Oct 21, 2024, 03:17 PM UTC
We are continuing to investigate this issue.
- monitoring Oct 21, 2024, 03:50 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 21, 2024, 04:12 PM UTC
This incident has been resolved.
- postmortem Oct 21, 2024, 09:05 PM UTC
Between 5:45 AM PDT and 7:45 AM PDT the SCORM Rustici service experienced a minor increase in error rates, followed by a more severe outage between 7:45 AM PDT and 8:45 AM PDT, after which point service was restored. The root cause of this outage was determined to be a misconfiguration in the internal load balancer, which resulted in general Rustici traffic routing to a single node and degraded performance when traffic exceeded a critical threshold. The infrastructure team has applied a fix as of the resolution of this outage and confirmed that traffic is correctly routing to all available nodes.