Jobylon incident
Overloaded queue causing error messages to applicants
Jobylon experienced a notice incident on December 25, 2020, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jan 13, 2021, 02:42 PM UTC
On 23/12 a faulty path was applied, which started spawning messages on our async queue. This kept on running through the 24-25/12 when it finally took down our queue. During the night between the 25/12 and 26/12 the queue was brought back up by our engineering team and a path with a fix was applied. Candidates who was applying during the period had sporadic issues and when the queue was brought down, all submitted applications was met with a error message. The applications was not lost and they could later be recovered.
- postmortem Jan 13, 2021, 02:43 PM UTC
To prevent this from happening again, we have now updated our alarms that notify the engineering team when the queue is under a heavy load.