Modo Labs experienced a major incident on December 9, 2019 affecting Modo Application Servers, lasting 10h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 09, 2019, 04:15 PM UTC
We are currently investigating an issue affecting some applications
- identified Dec 09, 2019, 05:26 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Dec 09, 2019, 06:35 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 10, 2019, 02:43 AM UTC
This incident has been resolved.
- postmortem Dec 11, 2019, 04:40 PM UTC
On December 9th, 2019 at approximately 15:42 UTC, Modo Operations staff were alerted to spikes in website performance for a number of sites. Investigation revealed a software issue causing anomalous database resource usage for which a fix was developed and deployed to appropriate sites at approximately 18:35 UTC. After monitoring system resources, the incident was resolved . No data was inadvertently released, and the software patch should prevent this issue from recurring.