Mindtickle incident

The series details page on the learning site was inaccessible

Notice Resolved View vendor source →

Mindtickle experienced a notice incident on December 29, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Dec 29, 2023, 02:36 AM UTC
Resolved
Nov 28, 2023, 05:54 PM UTC
Duration
—
Detected by Pingoru
Dec 29, 2023, 02:36 AM UTC

Update timeline

  1. resolved Dec 29, 2023, 02:36 AM UTC

    The series details page on the learning site was inaccessible. The end users could not access the Series to complete their programs.

  2. postmortem Dec 29, 2023, 02:36 AM UTC

    **Impact:** The series details page on the learning site was inaccessible for 22 minutes, from 23:24 PT to 23:46 PT on 28th November, 2023. **Why it happened:** A state mismatch occurred due to a serial deployment of two applications, leading to GraphQL queries failing. Incident timeline \(PT\): * 28th November, 2023 - 23:09 - Deployment triggered * 28th November, 2023 - 23:23 - Deployment partially completed * 28th November, 2023 - 23:24 - Spike in Errors observed * 28th November, 2023 - 23:40 - By this time, build was completed. We waited for the application to deploy. * 28th November, 2023 - 23:43 - Deployment completed * 28th November, 2023 - 23:46 - 400 Error rate dropped. **What we did to fix it:** Corrective steps: The state mismatch was resolved once the deployments were completed, and no further corrective action was necessary. **Preventive steps:** We defined the order of operations in deployment scripts to prevent a recurrence of this issue.