Technolutions incident
Degraded performance for databases on OPUS cluster
Technolutions experienced a notice incident on January 19, 2022 affecting Slate, lasting 5h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 19, 2022, 02:36 PM UTC
We are currently investigating degraded performance for databases on the OPUS cluster.
- investigating Jan 19, 2022, 04:38 PM UTC
Following the initial post, we throttled back some background jobs (rule execution and scheduled jobs) and have observed no further performance issues for the past two hours. We are slowly increasing the throughput of these background jobs to observe the effect on system performance. We do not have an identified root cause at this time and are exploring the possibility of an issue with underlying hardware or infrastructure.
- resolved Jan 19, 2022, 07:47 PM UTC
No further performance issues have been observed, so we will proceed with resolving this incident. We will continue to monitor these systems and should any performance issues re-occur, we will proceed with additional mitigations we have prepared for an overnight window.