ThinkLink experienced a minor incident on August 20, 2021 affecting ThinkLink Platform, lasting 7d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 20, 2021, 06:01 PM UTC
Our database is experiencing intermittent CPU usage spikes. When these spikes occur, users may experience "database exception" error message. Normal performance is restored as a spike subsides. We are actively in the process of increasing our database capacity so that the spikes will consume a lower percentage of available resources. We will send an update when this work is complete.
- monitoring Aug 20, 2021, 06:22 PM UTC
The capacity of our database has been increased and usage has returned to normal levels. We are continuing to monitor performance live and will make further adjustments as needed.
- identified Aug 23, 2021, 05:14 PM UTC
We are currently experiencing a high incidence of database errors, some of which are preventing users from logging in to the ThinkLink system. Our application team is actively looking at database connections and a restart may be required. We will update this incident as soon as new information becomes available.
- identified Aug 23, 2021, 05:18 PM UTC
Our application team has identified that there are several login requests that are "stuck." The team is terminating these processes to free up capacity on the database. Some users who have been attempting to log in may need to log in again once the processes are ended. We are continuing our investigation to prevent this issue going forward.
- monitoring Aug 23, 2021, 05:42 PM UTC
The problem processes have been terminated and the database has recovered. Users should now be able to log in normally. We are continuing to monitor the database for issues and will keep this incident updated.
- monitoring Aug 24, 2021, 02:33 PM UTC
We are continuing to monitor the intermittent database exception errors that have been presenting to some users. Last night we make some signficant network hardware upgrades and added database indexing to reduce the run time of certain processes that were tying up the system. Our application team is monitoring database performance and implementing additional indexing as needed on a daily basis.
- monitoring Aug 25, 2021, 06:30 PM UTC
Additional database indexing was added this morning and database consumption is well within expected ranges. Some users are still reporting occasional Database Exception errors that are dismissable and are not blocking functionality. We are investigating these lingering errors and will continue to update until resolved. We thank you for your continued patience as ThinkLink grows through its first back to school season.
- monitoring Aug 26, 2021, 03:28 PM UTC
Further indexing was added to ThinkLink this morning as our application team continues to chase down processes that are taking too long to run. Users should see a dramatic decrease in the frequency of "Database Exception" errors from Monday as we continue to add this indexing. Our servers are continuing to perform as expected today and we are actively monitoring should anything change.
- resolved Aug 27, 2021, 07:09 PM UTC
We are pleased to report that we are on day three of healthy database performance. We are aware of a handful of pages that may be triggering errors and are adding indexing to our database to address those. Please contact our technical support team at [email protected] with the name of the page or the action during which the error presented so that we can track these instances. We thank you all for your patience as we worked through the issue.