12Twenty experienced a notice incident on September 30, 2020, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 15, 2020, 08:48 PM UTC
Background jobs processed through the 12Twenty platform are experiencing delays. Background jobs include the sending of emails (including email verification codes), document packet generation, data uploads, etc. As an example, when generating document packets, the user will see a ‘please wait' until the background job infrastructure is able to finish processing the jobs queued before the user's job.
- postmortem Oct 15, 2020, 08:49 PM UTC
## Summary of Impact On 2020-09-30, between 9:00am PT and 12:30pm PT, background jobs processed through 12Twenty's platform were delayed. Background jobs include the sending of emails \(including email verification codes\), document packet generation, data uploads, etc. As an example, when generating document packets, the user saw a ‘please wait' until our background job infrastructure was able to finish processing the jobs queued before the user’s job. ## Root Cause A number of personalized mass emails with large attachments were sent by numerous schools within a short timeframe through our platform. Due to the attachments, the processing of each email took ~30x longer than emails normally take to process. 12Twenty did not sufficiently prioritize the various types of background jobs that run through the platform. ## Mitigation The background job infrastructure self-recovered after a little over 3 hours of delays. ## Next Steps The below highlights some of the steps we are taking to ensure this type of incident does not occur in the future: * Ability to easily increase the number of background job worker processes * Improve the prioritization of background job types * Faster scale out of background job infrastructure * Work with our email sending provider to look into the excessive performance degradation when emails with large attachments are sent