Technolutions incident
Background worker processing delayed overnight
Technolutions experienced a notice incident on March 17, 2020 affecting Slate, lasting 6h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 17, 2020, 11:41 AM UTC
Background worker processing was delayed overnight. All jobs, including imports, scheduled exports, mailings, and rules, are running at this time, and scheduled exports will be completing shortly. We're investigating the conditions that caused background worker node capacity to be substantially reduced.
- resolved Mar 17, 2020, 06:14 PM UTC
We believe that we have identified and accounted for the source of the issue that reduced system capacity for background worker processing, where a software update triggered a partial shutdown of worker nodes that were not subsequently and automatically restarted. This reduced system capacity delayed the execution of evening and overnight jobs, including imports, scheduled exports, mailings, and rules. Tonight, prior to the overnight delivery window, we will be confirming that a reproduction of these conditions does not again reduce system capacity. We will continue to monitor all system processes to ensure that system capacity remains fully online and operational.