Catalyst Cloud incident

Network hardware issue porirua

Major Resolved View vendor source →

Catalyst Cloud experienced a major incident on October 5, 2024 affecting Compute (nz-por-1) and Block storage (nz-por-1) and 1 more component, lasting 6h 59m. The incident has been resolved; the full update timeline is below.

Started
Oct 05, 2024, 09:13 PM UTC
Resolved
Oct 06, 2024, 04:13 AM UTC
Duration
6h 59m
Detected by Pingoru
Oct 05, 2024, 09:13 PM UTC

Affected components

Compute (nz-por-1)Block storage (nz-por-1)Network (nz-por-1)

Update timeline

  1. investigating Oct 05, 2024, 09:13 PM UTC

    We are experiencing a hardware issue with our networking fabric in the nz-por-1 region. Some services might experience degraded performance.

  2. identified Oct 05, 2024, 09:35 PM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Oct 05, 2024, 10:19 PM UTC

    We are working on replacing the network hardware but continue to experience issues with hdd block storage in porirua

  4. identified Oct 05, 2024, 10:53 PM UTC

    We are about to replace the broken hardware. In the meantime a mitigation is in place that has greatly reduced the risk of outages.

  5. identified Oct 06, 2024, 02:27 AM UTC

    We are still working on the replacement hardware.

  6. resolved Oct 06, 2024, 04:13 AM UTC

    We have replaced the failed hardware. Full service has returned.