Learnosity incident
Investigating possible scoring delays in US-East-1 Region
Learnosity experienced a minor incident on April 26, 2023 affecting Availability of session information and Branching/Adaptive services, lasting 1h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 26, 2023, 01:42 PM UTC
As of 9:00 am EDT, 1:00 pm UTC, we are investigating possible scoring delays of submitted assessments in the US-East-1 region. Authoring and assessment functionality remains largely unaffected, and no data loss is evident. If confirmed, impact will be on availability of submitted sessions via the Data API, and possibly adaptive testing due to the need for per-item scoring Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
- investigating Apr 26, 2023, 02:05 PM UTC
As of 10:00 am EDT, 2:00 pm UTC, we are still investigating possible scoring delays of submitted assessments and have not confirmed any direct customer impact. As a precautionary measure, we will continue to show degraded performance for availability of session information until we're certain no possibility for delays exists. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
- investigating Apr 26, 2023, 02:24 PM UTC
As of 10:20 am EDT, 2:20 pm UTC, we are still investigating possible scoring delays of submitted assessments and have confirmed that a small number of customers are experiencing delays in availability of session information. One of several scoring queues is suffering degraded performance and is recovering, and sessions are moving through other queues, with backlog reducing at a substantial rate. All other APIs remain operational, there is no known data loss, and only a small subset of customers have been affected. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
- monitoring Apr 26, 2023, 02:53 PM UTC
As of 10:45 am EDT, 2:45 pm UTC, the scoring backlog is now fully recovered and we have moved to monitoring. This update is mainly to note that our support ticket system, by our vendor Zendesk, is experiencing unrelated issues with loading tickets. Errors appear small in number and intermittent, but note that we have responded to all tickets received, including the two related to this incident. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
- resolved Apr 26, 2023, 03:20 PM UTC
As of 11:15 am EDT, 3:15 pm UTC, the issue causing delayed availability of a small subset of submitted sessions in us-east-1 has been resolved. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.