Knak experienced a minor incident on April 14, 2021 affecting Knak App, lasting 16h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 14, 2021, 07:43 PM UTC
We are currently investigating an incident where the index for emails and pages in studio has been lost. We are currently rebuilding it from the latest emails and pages backward. The emails and pages will still be able to be retrieved by ID using the URL, but will not be accessible from Studio until the index is rebuilt.
- investigating Apr 14, 2021, 07:46 PM UTC
We are continuing to investigate this issue.
- investigating Apr 14, 2021, 09:38 PM UTC
Emails and pages that have been update since the beginning of March have been reindexed.
- identified Apr 14, 2021, 09:39 PM UTC
The issue has been identified and a fix is being implemented.
- identified Apr 15, 2021, 11:46 AM UTC
We are continuing to work on a fix for this issue.
- identified Apr 15, 2021, 11:46 AM UTC
Index has been fully restored.
- resolved Apr 15, 2021, 11:46 AM UTC
This incident has been resolved.
- postmortem Apr 19, 2021, 04:13 PM UTC
# Context Knak uses Algolia for a few pages to make searching and finding information quicker and easier. The Studio page \(formerly Email Center\) relies on Algolia search capabilities both for displaying emails and landing pages and providing searchability on those assets. _**Note:** The email and page data is all stored in the database and was unaffected, and the index only stores a "summary" of the information to enable quicker searching._ # Issue Timeline April 14th, 2021 3:20PM During clean-up of Algolia, to remove data generated through our automated tests, our production index was cleared accidentally by an engineer. This made it such that the emails could no longer be found in the Email Center/Studio. The issue was flagged immediately and we put up a Status Page alert to explain the issue to clients. We then built out a script to re-index emails and landing pages, starting by the most recently updated. Within about 90 minutes, any emails that had been built or updated since March 1 had been restored in the index and could be found in the studio. Over the next few hours, we were able to restore all emails and pages created in Knak. # Path Forward **Separation of Applications** The indexes are currently all within one application. We will be separating out the production instance from the rest of the instances to remove the risk of accidental deletion. Anyone that has access to the production instances of Algolia will need to do so through a separate login and permissions on the production index will be very limited. **Daily Backup of index** We are putting measures in place to backup the Algolia index daily. Should we ever lose an index in the future, we would be able to restore an existing backup and only need to make updates on emails and pages that were updated on that day, making the recovery of a lost index take 10-15 minutes rather than a few hours.