Kahoot! experienced a critical incident on December 22, 2022 affecting Message sending, lasting 1h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 22, 2022, 11:41 AM UTC
Actimo - We are seeing issues with sendouts - at the moment, clients will not be able to send any messages via any delivery channel, we are on it with the Devs to see what is happening here and trying to get any queued messages sent once we have resolved the issue.
- identified Dec 22, 2022, 11:50 AM UTC
The issue has been identified and a fix is being implemented.
- identified Dec 22, 2022, 12:45 PM UTC
The database issue has been identified and we have escalated this to our hosting provider.
- identified Dec 22, 2022, 01:06 PM UTC
Rollback has been initiated. Processing the queues.
- identified Dec 22, 2022, 01:26 PM UTC
We need to restart hosting services in order to apply the changes. Apologies for the inconvenience and the app will be unavailable for a few minutes as a consequence
- resolved Dec 22, 2022, 01:39 PM UTC
The service is working as normal now and we will monitor the service for a post mortem.
- postmortem Dec 22, 2022, 02:51 PM UTC
**Incident summary:** One of the transactions in the DB put a lock on one of the main tables. **Description**: Inside the process of publishing a message one of the transactions hanged inside our DB server. Transaction showed resistance towards being killed manually or with any other attempts **Resolution:** DB’s writer instance was put into reboot to force clean running DB threads. Fix has been implemented for message dispatches to prevent this happening in the future