GUIDEcx experienced a minor incident on October 4, 2024, lasting 1h 35m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Oct 04, 2024, 04:15 PM UTC
We are currently investigating this incident..
- investigating Oct 04, 2024, 04:33 PM UTC
We are currently investigating this incident. The behavior we are seeing is that users access the Project page, click on a project and rather than being directed to the Plan view are redirected back to the Project page. .
- monitoring Oct 04, 2024, 04:44 PM UTC
We implemented a fix and are currently monitoring the result. The initial results are positive and it appears that access to projects has been restored. .
- resolved Oct 04, 2024, 04:52 PM UTC
This issue has been resolved. .
- resolved Oct 04, 2024, 05:50 PM UTC
## Root Cause Analysis ### **Issue Summary:** On the morning of October 4, 2024, at 10:00 AM MST, an error spike occurred on project plan loading after the release of an improved database view designed to enhance project statistics load time and overall database performance. The issue was resolved by 10:40 AM MST. ### **Root Cause:** The new database view was incompatible with old pods, which caused errors when they accessed the updated view. Our automated rollback process was triggered in response to the error spike, but it only rolled back the application deployment, not the database schema. As a result, all pods continued to access the incompatible view, extending the period of disruption. ### **Resolution:** At 10:30 AM MST, we redeployed the latest version of the application, ensuring that all pods were compatible with the new database view, resolving the issue. ### **Preventive Measures:** * Improve the canary release cycle to isolate better and test database changes before full production rollouts. * Enhance the automated rollback process to include database schema rollbacks when necessary..