Thought Industries incident

Rustici SCORM Outage (US)

Resolved after 1 hour 46 minutes, affecting US - Academy Platform, on October 21, 2024.

postmortem Oct 21, 2024, 09:05 PM UTC

Between 5:45 AM PDT and 7:45 AM PDT the SCORM Rustici service experienced a minor increase in error rates, followed by a more severe outage between 7:45 AM PDT and 8:45 AM PDT, after which point service was restored. The root cause of this outage was determined to be a misconfiguration in the internal load balancer, which resulted in general Rustici traffic routing to a single node and degraded performance when traffic exceeded a critical threshold. The infrastructure team has applied a fix as of the resolution of this outage and confirmed that traffic is correctly routing to all available nodes.

Started
Oct 21, 2024, 02:26 PM UTC
Resolved
Oct 21, 2024, 04:12 PM UTC
Duration
1h 46m
Detected by Pingoru
Oct 21, 2024, 02:26 PM UTC

Affected components

US - Academy Platform

Update timeline

  1. investigating Oct 21, 2024, 02:26 PM UTC

    We are investigating high loading times on Rustici SCORM launches in the US.

  2. investigating Oct 21, 2024, 03:17 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Oct 21, 2024, 03:50 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Oct 21, 2024, 04:12 PM UTC

    This incident has been resolved.

  5. postmortem Oct 21, 2024, 09:05 PM UTC

    Between 5:45 AM PDT and 7:45 AM PDT the SCORM Rustici service experienced a minor increase in error rates, followed by a more severe outage between 7:45 AM PDT and 8:45 AM PDT, after which point service was restored. The root cause of this outage was determined to be a misconfiguration in the internal load balancer, which resulted in general Rustici traffic routing to a single node and degraded performance when traffic exceeded a critical threshold. The infrastructure team has applied a fix as of the resolution of this outage and confirmed that traffic is correctly routing to all available nodes.

Your product runs on more than Thought Industries

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →