Learnosity incident
Issue affecting previews of newly authored content in the US-East-1 region
Learnosity experienced a minor incident on March 29, 2022 affecting Creating and saving of Items/Activities/Tags, lasting 1h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 29, 2022, 05:09 PM UTC
As of 16:45 UTC, we are seeing delays in availability of newly authored questions/items in the US-East-1 region. Content is being created and persisted without issue, and is being successfully replicated to the CA, OR, IE and AUS regions. There is no data loss, but authoring previews are delayed in the VA region. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.
- identified Mar 29, 2022, 05:22 PM UTC
As of 17:15 UTC, we have identified an issue with content replication in US-East-1 Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
- identified Mar 29, 2022, 06:00 PM UTC
As of 17:45 UTC, we've routed the content replication backlog to the US-West-1 (CA) region while we await additional assistance from AWS to restore the unresponsive server in US-East-1 (VA). Content continues to save without data loss, and continues to preview successfully in all regions worldwide, other than VA. Further, the backlog has already reduced by two-thirds and is still dropping. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
- monitoring Mar 29, 2022, 06:10 PM UTC
As of 18:00 UTC, preview/availability of newly authored content is functioning correctly, and the US-East-1 (VA) server has been restored. We will continue to monitor for a short period and then begin transitioning the replication load back to the restored VA server. Some latency may still be apparent until the load has been fully transitioned to VA. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.
- resolved Mar 29, 2022, 06:37 PM UTC
As of 18:30 UTC, after a post-resolution monitoring period, we have seen no additional issues with newly authored content becoming available in preview or assessment settings. We are resolving this issue but will continue to monitor for a brief follow-on period. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.