ThinkLink incident

Degraded Performance

Minor Resolved View vendor source →

ThinkLink experienced a minor incident on April 1, 2022 affecting ThinkLink Platform, lasting 2d 18h. The incident has been resolved; the full update timeline is below.

Started
Apr 01, 2022, 06:15 PM UTC
Resolved
Apr 04, 2022, 01:14 PM UTC
Duration
2d 18h
Detected by Pingoru
Apr 01, 2022, 06:15 PM UTC

Affected components

ThinkLink Platform

Update timeline

  1. investigating Apr 01, 2022, 06:15 PM UTC

    Some users are reporting getting a generic error message when attempting to manage courses. This appears to be tied to slowness with one of the ThinkLink application's microservices. We are engaging with our hosting partner to troubleshoot and restart the service as needed.

  2. identified Apr 01, 2022, 07:02 PM UTC

    Our hosting partner will be initiating rolling restarts of the ThinkLink application to restore performance. Users may be momentarily logged out during the restart process but should immediately be able to log back in. We will provide an update when restarts are complete.

  3. identified Apr 01, 2022, 07:48 PM UTC

    Rolling restarts have been completed and users are still experiencing errors in parts of the application. We are continuing to diagnose the issue.

  4. identified Apr 03, 2022, 10:20 PM UTC

    The issue has been narrowed down to an expired access token for the file storage system and we are working with our hosting partner to renew the token. Once the token has been replaced we will need to complete another round of rolling restarts which will resolve the issues. We thank you for your patience.

  5. monitoring Apr 04, 2022, 12:46 PM UTC

    Access tokens have been updated and the application is being restarted. Users may be logged out of the system but should immediately be able to log back in. This restart will restore full functionality and we are monitoring the application to confirm.

  6. resolved Apr 04, 2022, 01:14 PM UTC

    The incident has been resolved and ThinkLink has returned to normal functionality. We thank you for your patience while we resolved this issue.