ThinkLink incident

ThinkLink application - Database Exception Errors

Minor Resolved View vendor source →

ThinkLink experienced a minor incident on November 17, 2021 affecting ThinkLink Platform, lasting 1d 16h. The incident has been resolved; the full update timeline is below.

Started
Nov 17, 2021, 01:17 AM UTC
Resolved
Nov 18, 2021, 05:18 PM UTC
Duration
1d 16h
Detected by Pingoru
Nov 17, 2021, 01:17 AM UTC

Affected components

ThinkLink Platform

Update timeline

  1. investigating Nov 16, 2021, 04:57 PM UTC

    We have had reports of database exception errors and slow response times. We do not see any issues in the system logs but we do have the vendor reviewing the application to determine root cause.

  2. monitoring Nov 16, 2021, 05:28 PM UTC

    The issue appears to have cleared. We are monitoring it for a time.

  3. monitoring Nov 16, 2021, 09:11 PM UTC

    The issue is still happening periodically. We have reached out to vendor support and they have started looking at the database issue.

  4. monitoring Nov 17, 2021, 01:17 AM UTC

    The last Azure outage with the database was resolved at 4:14pm ET. Our hosting vendor changed the hardware on our database at 7:45pm ET. While we expect that this has resolved the database unavailability we are continuing to monitor for any further issues.

  5. investigating Nov 17, 2021, 08:11 PM UTC

    Despite the change to Azure hardware last night, issues with errors and slowness are persisting. We are actively working with our hosting and application teams to determine the root cause and resolve the issue. Thank you for your patience.

  6. investigating Nov 17, 2021, 09:11 PM UTC

    We will be performing rolling restarts of our servers shortly to attempt to resolve potential stuck processes that could be causing the network issues. Users logged into the system may be logged out but should be able to log back into the system right away.

  7. monitoring Nov 17, 2021, 09:33 PM UTC

    Rolling restarts have been completed and we are monitoring the system.

  8. monitoring Nov 18, 2021, 01:35 PM UTC

    We are continuing to monitor for any further issues.

  9. resolved Nov 18, 2021, 05:18 PM UTC

    This weeks incident with ThinkLink errors and slowness is now confirmed to be resolved. The restart of our servers yesterday afternoon has cleared the stuck processes and system functionality has returned to normal. Thank you for your patience.