GUIDEcx experienced a minor incident on October 4, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 04, 2024, 04:15 PM UTC
Type: Incident Duration: 37 minutes Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking Oct 4, 16:15:26 GMT+0 - Investigating - We are currently investigating this incident. Oct 4, 16:33:40 GMT+0 - Investigating - We are currently investigating this incident. The behavior we are seeing is that users access the Project page, click on a project and rather than being directed to the Plan view are redirected back to the Project page. Oct 4, 16:44:59 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. The initial results are positive and it appears that access to projects has been restored. Oct 4, 16:52:54 GMT+0 - Resolved - This issue has been resolved. Oct 4, 17:50:47 GMT+0 - Resolved - ## Root Cause Analysis ### **Issue Summary:** On the morning of October 4, 2024, at 10:00 AM MST, an error spike occurred on project plan loading after the release of an improved database view designed to enhance project statistics load time and overall database performance. The issue was resolved by 10:40 AM MST. ### **Root Cause:** The new database view was incompatible with old pods, which caused errors when they accessed the updated view. Our automated rollback process was triggered in response to the error spike, but it only rolled back the application deployment, not the database schema. As a result, all pods continued to access the incompatible view, extending the period of disruption. ### **Resolution:** At 10:30 AM MST, we redeployed the latest version of the application, ensuring that all pods were compatible with the new database view, resolving the issue. ### **Preventive Measures:** * Improve the canary release cycle to isolate better and test database changes before full production rollouts. * Enhance the automated rollback process to include database schema rollbacks when necessary.