12Twenty experienced a notice incident on October 15, 2020, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 15, 2020, 08:55 PM UTC
Background jobs processed through the 12Twenty platform are experiencing delays. Background jobs include the sending of emails (including email verification codes), document packet generation, data uploads, etc. As an example, when generating document packets, the user will see a ‘please wait' until the background job infrastructure is able to finish processing the jobs queued before the user's job.
- postmortem Oct 15, 2020, 08:56 PM UTC
## **Summary of Impact** On 2020-10-07, between 11:20am PT and 1:30pm PT, background jobs processed through 12Twenty's platform were delayed. Background jobs include the sending of emails \(including email verification codes\), document packet generation, data uploads, etc. As an example, when generating document packets, the user saw a ‘please wait' until our background job infrastructure was able to finish processing the jobs queued before the user’s job. ## **Root Cause** An Azure service issue caused Office 365 calendar sync jobs to freeze up. These jobs stayed in a ‘processing’ state for over 2 hours and prevented other background jobs from being queued. From Azure: > Summary of Impact: Between 18:20 UTC and 18:42 UTC on 07 Oct 2020, a subset of customers may have encountered increased latency, packet loss, failed connections and authentication failures across multiple Azure services. Retries may have succeeded during this time and users who had authenticated prior to the impact start time were less likely to experience authentication issues. ## **Mitigation** Background job services were restarted around 1:00pm PT. The background job infrastructure took about 30 minutes to finish processing the delayed jobs in the queue. ## **Next Steps** The below highlights some of the steps we are taking to ensure this type of incident does not occur in the future: * Improve monitoring of background job infrastructure to ensure we are alerted in a more timely manner if delays are occurring * Determine why connectivity to Azure caused the jobs to hang and ensure that similar issues do not prevent other jobs from processing in the future