Modo Labs experienced a major incident on July 7, 2019 affecting Modo App Center, lasting 1h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 07, 2019, 04:14 PM UTC
Modo is working with a database hosting provider to fix an upgrade during the maintenance which encountered some issues. Application hosting is not impacted; however editing and changing pages may not be available. We will update this page when more information is available.
- resolved Jul 07, 2019, 05:37 PM UTC
The database which was affecting App Center Admin functions has been restored to full functionality.
- postmortem Jul 08, 2019, 02:15 PM UTC
During the maintenance window, an admin database had a version upgrade. The upgrade became wedged, and took several times longer than other upgrades. After entering a support ticket at the vendor, the issue was resolved at 1:07pm EDT. A non-admin database was also upgraded, and while its upgrade proceeded smoothly, shortly thereafter we encountered an issue with high memory utilization. This issue had been encountered previously after a back-end vendor patch. A set of specific memory utilization parameters were applied and the database instance returned to nominal status without errors by 2pm EDT.