Technolutions incident
Increased latency with PDF rasterization
Technolutions experienced a notice incident on January 2, 2020 affecting Slate, lasting 1d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 02, 2020, 04:36 PM UTC
We are investigating increased latency this morning with PDF rasterization, which is used to generate documents for display in the Slate Reader. This appears to be a result of increased CPU utilization on the Redis caching servers. We are actively working to determine the underlying cause of this increased CPU utilization.
- monitoring Jan 02, 2020, 08:07 PM UTC
We are continuing to investigate the root cause behind increased CPU utilization on the Redis caching servers earlier this morning. PDF rasterization was operating within normal latency parameters by approximately 12:30pm Eastern Time. We will continue to monitor these caching servers as we complete our investigation.
- resolved Jan 03, 2020, 09:43 PM UTC
The high CPU utilization experienced on Thursday morning has not reoccurred, and additional optimizations implemented on Thursday evening have substantially improved connection reuse, which we determined to be a contributory factor in the Thursday morning event. Rendering times today saw a median execution time of 75ms and a mean of 100ms, well within desired target ranges for the approximately 5 million invocations of the PDF rasterization service today. We will continue to monitor all infrastructure to ensure that all systems are performing within our desired target ranges.