Manchester Datacentre Performance issues

Update timeline

1. identified Apr 14, 2023, 01:59 PM UTC

   Please note we're aware of performance issues with relation to Accessing of services and are working to mitigate the problem.

2. monitoring Apr 14, 2023, 02:22 PM UTC

   Performance issues should now be returning to normal. Our engineer on site has replaced a faulty component and the sync process between peered hardware components appears to of completed successfully. Some isolated cases of restarting servers may be required to regain resilience but those clients affected will be contacted shortly.

3. resolved Apr 14, 2023, 03:10 PM UTC

   We're now marking this incident as resolved as restart times have been agreed for those clients requiring reboots for resilient pathing.

4. postmortem Apr 14, 2023, 03:10 PM UTC

   **Incident:** During the replacement of a datacentre hardware component clients encountered a brief interruption of service \(<1minute\) followed by a small window of degraded performance \(~20 mins\) . This issue was caused by the component taking a slightly longer time than expected to sync up with its peer device and a brief lock up of the unit itself. These modules operate in a mirror format so that should one component fail in its entirety the second picks up the slack and continues on processing the request. In this instance it was the syncing process that brings the new device “up to speed” that actually caused the problem. ‌ **Resolution:** Engineers resolved the issue by manually performing a secondary sync of the unit from the user interface and allowing the sync process to complete. ‌ **Take Away Actions:** While the unit should of synced seamlessly without incident and has indeed done so in the past \(per manufacturers guidelines\), we will endeavour to adjust our maintenance window timings for this component in future. We will review this brief outage with the vendor of the component to see if further mitigation steps could of been applied.