Schoolyear experienced a minor incident on March 19, 2025 affecting Safe Exam Workspace, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Mar 19, 2025, 09:24 AM UTC
On the 17th and 18th of March, mainly around 11:00 UTC, we saw an increase in request failures, meaning a small portion of the requests made to our servers resulted in a 502 Bad Gateway error. This resulted in some students seeing an error in our application or in that of applications with a Schoolyear integration. For those impacted, estarting our application or refreshing the page solved the problem. For the majority of the users, there was no impact and they were able to use our software without problems. This incident was caused by a faulty background task that crashed the server that performed the task. Our backend is designed to handle such failures, as unhealthy servers are detected and redeployed automatically, limiting the impact to only a few requests that can simply be retried. In this case, however, each server that tried to perform the background task crashed, resulting in a cascade of failing servers, which became noticible. As the relevant background task was only retried 3 times, this incident did not result in an outages, as there were enough servers in the cluster still available to handle incoming requests. After being alerted of the failures by one of our integration partners, we found the failing background task and fixed the failing task. Additionally, we deployed a mechanism to prevent any faulty background task from crashing a server in the future. As these crashes never resulted in an outage, the impact was limited. Nonetheless, the fact that we did not detect these failures ourself immediately will be evalutated by our operations team.