MerchantSpring incident
Partial database outage on Analytics Dashboard
MerchantSpring experienced a major incident on December 21, 2022 affecting Analytics Dashboard and Data Pipeline, lasting 16h 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 21, 2022, 10:07 AM UTC
One of our production databases has suffered an unexpected outage, and we are in the process of restoring the database from the latest set of backups and write-ahead logs. We do not anticipate any data to be lost as a result of this incident. During this time, some users may not able to log in, and some widgets may fail to load. We expect the database restoration process to be completed within the next 24 hours. Once this is done, we will commence the backfilling of marketplace data to ensure there are no gaps in the data.
- identified Dec 21, 2022, 10:07 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Dec 21, 2022, 05:38 PM UTC
The database restoration process has been completed, and all systems are operating normally. Backfilling of marketplace data has begun, and we are continuing to monitor the health of the platform.
- resolved Dec 21, 2022, 11:53 PM UTC
We have successfully completed the backfilling of marketplace data covering the period affected by the database outage. This incident is now fully resolved. We apologise for the inconvenience, and will be working with our database provider to determine the cause of the outage.