Learnosity incident

Increased errors and degraded performance in US-east-1

Major Resolved View vendor source →

Learnosity experienced a major incident on December 22, 2021 affecting Loading and rendering of assessment UI and Loading and rendering of reports and 1 more component, lasting 3h 18m. The incident has been resolved; the full update timeline is below.

Started
Dec 22, 2021, 12:56 PM UTC
Resolved
Dec 22, 2021, 04:14 PM UTC
Duration
3h 18m
Detected by Pingoru
Dec 22, 2021, 12:56 PM UTC

Affected components

Loading and rendering of assessment UILoading and rendering of reportsCreation of sessionsLoading and rendering of Items/Questions/FeaturesAvailability of session informationCreation of Item PoolsSaving of student responsesTag-based reportsUpdating session response scoresBranching/Adaptive services

Update timeline

  1. investigating Dec 22, 2021, 12:56 PM UTC

    As of 04:35 AM PST AWS has been experiencing problems that are trickling down to Learnosity. It appears that some areas of the U.S. are having trouble accessing some data and some assessments may be affected. Learnosity systems are running impaired and we will update this status as more information becomes available.

  2. identified Dec 22, 2021, 01:14 PM UTC

    AWS have experienced an outage in one Availability Zone and it's affecting the systems running in that AZ. Most of Learnosity systems are still operational but this AWS outage is having a knock-on effect that is impacting other AWS systems.

  3. monitoring Dec 22, 2021, 02:03 PM UTC

    Looking like AWS have resolved the issue on their end, our systems are starting to stabilise and looking to be in good health again. We will continue to monitor the situation and updated as needed. The error rates have dropped again to zero and all systems are looking green again.

  4. resolved Dec 22, 2021, 04:14 PM UTC

    As of 16:00 UTC, we are still seeing no lingering effects from this mornings AWS outage and are resolving this issue. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns.

  5. postmortem Dec 22, 2021, 04:15 PM UTC

    AWS experienced an outage in one Availability Zone in US-East-1 due to a power failure. Initially unaffected, Learnosity experienced some knock-on effects s few hours later. AWS has resolved the power issue and things quickly returned to normal.