Searchspring experienced a minor incident on January 26, 2022 affecting Indexing, lasting 2h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 26, 2022, 03:04 PM UTC
We are currently investigating reports of slower than normal data generation during indexing. It does appear we have a larger than normal queue and are exploring possible causes.
- identified Jan 26, 2022, 03:21 PM UTC
We have identified that some of our regeneration servers appear to not be running properly causing a longer than normal queue time. We are working on a fix to address this issue.
- identified Jan 26, 2022, 03:44 PM UTC
We have found an issue that has introduced some bad data into our database for indexing configuration. We are working on a fix to stop the bad data from being added to the database and then will proceed to fixing the bad data within the database.
- monitoring Jan 26, 2022, 04:13 PM UTC
Fixes have been deployed and they appear to have resolved the issue. Due to a longer than normal queue indexing may still be delayed, but we should work through the queue quickly.
- monitoring Jan 26, 2022, 04:15 PM UTC
A fix to the bad data has been implemented. The indexing queue is moving along and we will continue to monitor this issue.
- monitoring Jan 26, 2022, 04:16 PM UTC
We are continuing to monitor for any further issues.
- resolved Jan 26, 2022, 05:38 PM UTC
This incident has been resolved.