Mural experienced a major incident on February 18, 2022 affecting Exports and Notifications and 1 more component, lasting 2d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 18, 2022, 10:45 PM UTC
We are currently experiencing a service degradation that is affecting MURAL’s dashboard search functionality. Additionally, some members are not receiving password reset emails, confirmation emails, or export emails from the app. This should not affect general usage in the MURAL application. Our team is currently investigating and we will post another update when we have new information to share. We apologize for any inconveniences caused.
- investigating Feb 18, 2022, 11:45 PM UTC
We are continuing to investigate this issue. We will keep this site updated as soon as more information becomes available. We appreciate your patience and apologize for any inconvenience.
- identified Feb 19, 2022, 12:08 AM UTC
We have identified the root cause of these issues and implemented a fix that should allow password reset emails, confirmation emails, and export emails sent from the app to be fully functional again. We will continue to monitor the status of these and update this site as soon as we have more information to share. We are continuing to investigate the best way to implement a fix for the Search bar functionality. We will continue to update this site as soon as we have more information to share.
- identified Feb 19, 2022, 01:09 AM UTC
We have verified the fix implemented has allowed password reset emails, confirmation emails, and export emails sent from the app to be fully functional again. We are continuing to investigate the best way to implement a fix for the Search bar functionality. We will continue to update this site as soon as we have more information to share.
- identified Feb 19, 2022, 03:42 PM UTC
The fix we implemented for password reset emails, confirmation emails and export emails is stable and this functionality is working correctly. Search functionality remain restricted. Searching for murals using keywords will not currently return any results. Our engineers have identified the path to resolving this issue and are working diligently to implement an update as quickly as possible. We expect to have further details within the next 24 hours. In the meantime you can continue to use the 'your recently opened' list to see your most current murals, or search visually within your dashboard. We apologize for this ongoing inconvenience and appreciate your understanding and patience.
- monitoring Feb 21, 2022, 04:39 AM UTC
We are currently monitoring the fix we implemented for password reset emails, confirmation emails, and export emails. The fix that has been deployed is stable and functionalities are working properly. The search functionality is already working for all workspaces. For globalTemplates in every workspace, those currently remain searchable through the "create mural" option. We appreciate your understanding and patience.
- resolved Feb 21, 2022, 05:41 AM UTC
Password reset emails, confirmation emails, export emails, and search functionality have been restored. For template search please use the create mural option. We appreciate your understanding and patience in this matter.
- postmortem Mar 02, 2022, 05:56 PM UTC
**What happened?** On Friday, February 18, 2022, the search engine in MURAL was upgraded from ElasticSearch 6 to ES7. The change proceeded as planned and no immediate issues were detected during our post-release checks. Later that day our engineers detected unexpected behavior with MURAL search results. An immediate decision was taken to roll back to the previous version. The process to roll back the upgrade was unsuccessful. Our team worked through the weekend to restore search functionality within MURAL workspaces. Public API endpoints for searching were however inoperative. Our team continued working through the next week to identify the root of the issue with ElasticSearch. On Friday, February 25, the indexes were fully rebuilt on a new ES6 cluster, tested successfully in beta, and then enabled in production. This restored full functionality using ES6, including the public API endpoints for search. **Summary** This incident resulted in a partial outage for search functionality within MURAL workspaces, lasting 54 hours and 56 minutes. MURAL remained operational throughout this incident and members were able to log in and access their data. **What we've done to avoid this happening again** We are re-evaluating our upgrade process for ES6 to ES7 to ensure this does not happen again during the next upgrade.