Probado incident

Datacentre Component failure - Follow up Maintenance

Minor Resolved View vendor source →

Probado experienced a minor incident on August 3, 2022 affecting VoIP (Voice over IP) and Data Services (Hosting), lasting 9h 44m. The incident has been resolved; the full update timeline is below.

Started
Aug 03, 2022, 09:40 AM UTC
Resolved
Aug 03, 2022, 07:25 PM UTC
Duration
9h 44m
Detected by Pingoru
Aug 03, 2022, 09:40 AM UTC

Affected components

VoIP (Voice over IP)Data Services (Hosting)

Update timeline

  1. identified Aug 03, 2022, 09:40 AM UTC

    We've identified an issue with a hardware component within the datacentre. Engineers have arrived on site to replace the faulty component. Resilient network configuration means client's servers should be unaffected during this outage but may need to be restarted to ensure full resilience is restored once the faulty component has been replaced.

  2. identified Aug 03, 2022, 11:56 AM UTC

    The replacement component has now been installed. We've scheduled a restart of host servers at 7pm tonight to restore resilient pathing. Please note your servers will be unavailable during this time while we perform these essential restarts

  3. resolved Aug 03, 2022, 07:25 PM UTC

    Maintenance is now complete and all servers should be available for use. Many thanks for your patience

  4. postmortem Aug 03, 2022, 07:25 PM UTC

    **Overview:** A Hardware component failed within one of the Storage platforms within the Datacentre. As we offer resilient technologies this meant that affected client servers continued to function. **Resolution:** We sourced a replacement part from Dell and upon arrival we dispatched an engineer to install the part. Once physically installed we waited out of core business hours to perform configuration tasks and ensure that the servers could see primary and secondary connections again restoring the resiliency. ‌ **Outcome from Incident:** The Technology we offer within the Datacentre performed to spec allowing the servers to continue their operations despite hardware failure. Replacing the part and performing the aforementioned configuration tasks will cater for this scenario should hardware failure occur again.