Catalyst Cloud Outage History

Catalyst Cloud is up right now

Catalyst Cloud had 40 outages in the last 2 years totaling 184h 26m of downtime — averaging 1.6 incidents per month.

There were 40 Catalyst Cloud outages since June 25, 2024 totaling 184h 26m of downtime. Each is summarised below — incident details, duration, and resolution information.

Source: https://status.catalystcloud.nz

Major October 5, 2024

Network hardware issue porirua

Detected by Pingoru
Oct 05, 2024, 09:13 PM UTC
Resolved
Oct 06, 2024, 04:13 AM UTC
Duration
6h 59m
Affected: Compute (nz-por-1)Block storage (nz-por-1)Network (nz-por-1)
Timeline · 6 updates
  1. investigating Oct 05, 2024, 09:13 PM UTC

    We are experiencing a hardware issue with our networking fabric in the nz-por-1 region. Some services might experience degraded performance.

  2. identified Oct 05, 2024, 09:35 PM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Oct 05, 2024, 10:19 PM UTC

    We are working on replacing the network hardware but continue to experience issues with hdd block storage in porirua

  4. identified Oct 05, 2024, 10:53 PM UTC

    We are about to replace the broken hardware. In the meantime a mitigation is in place that has greatly reduced the risk of outages.

  5. identified Oct 06, 2024, 02:27 AM UTC

    We are still working on the replacement hardware.

  6. resolved Oct 06, 2024, 04:13 AM UTC

    We have replaced the failed hardware. Full service has returned.

Read the full incident report →

Minor October 5, 2024

Reduced b1.standard block storage performance in nz-por-1

Detected by Pingoru
Oct 05, 2024, 01:25 PM UTC
Resolved
Oct 05, 2024, 02:34 PM UTC
Duration
1h 9m
Affected: Block storage (nz-por-1)
Timeline · 3 updates
  1. investigating Oct 05, 2024, 01:25 PM UTC

    We are currently investigating this issue.

  2. monitoring Oct 05, 2024, 01:52 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Oct 05, 2024, 02:34 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor September 16, 2024

Networking connectivity of GPU compute servers

Detected by Pingoru
Sep 16, 2024, 06:18 AM UTC
Resolved
Sep 17, 2024, 12:07 AM UTC
Duration
17h 48m
Affected: Compute (nz-por-1)
Timeline · 2 updates
  1. investigating Sep 16, 2024, 06:18 AM UTC

    Following an upgrade of the compute service, there is an issue that effects the network connectivity of GPU compute servers. All GPU servers will be prevented from having network access. Please note that other compute flavors are not affected.

  2. resolved Sep 17, 2024, 12:07 AM UTC

    This issue has been resolved - GPU instances are now functioning normally.

Read the full incident report →

Minor September 13, 2024

Billing services unavailable

Detected by Pingoru
Sep 13, 2024, 12:11 AM UTC
Resolved
Sep 13, 2024, 12:58 AM UTC
Duration
47m
Affected: Dashboard
Timeline · 3 updates
  1. identified Sep 13, 2024, 12:11 AM UTC

    The issue has been identified and a fix is being implemented.

  2. monitoring Sep 13, 2024, 12:27 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Sep 13, 2024, 12:58 AM UTC

    This incident has been resolved.

Read the full incident report →

Major August 18, 2024

NVME Block Storage Performance Degradation Porirua

Detected by Pingoru
Aug 18, 2024, 04:08 AM UTC
Resolved
Aug 18, 2024, 05:03 AM UTC
Duration
55m
Affected: Block storage (nz-por-1)
Timeline · 3 updates
  1. identified Aug 18, 2024, 04:08 AM UTC

    The issue has been identified and a fix is being implemented.

  2. monitoring Aug 18, 2024, 04:32 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Aug 18, 2024, 05:03 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor August 11, 2024

Block storage degraded performance in Hamilton

Detected by Pingoru
Aug 11, 2024, 02:40 AM UTC
Resolved
Aug 11, 2024, 04:08 AM UTC
Duration
1h 28m
Affected: Block storage (nz-hlz-1)
Timeline · 2 updates
  1. monitoring Aug 11, 2024, 02:40 AM UTC

    We have had an incident with block storage in Hamilton. We have implemented a fix and are monitoring the result. The symptoms would have been very long access times on nvme volumes.

  2. resolved Aug 11, 2024, 04:08 AM UTC

    Our fix was succesfull and volumes have been behaving normally since 14:30. We are monitoring for any further issues.

Read the full incident report →

Notice August 8, 2024

Block Storage Maintenance in Porirua

Detected by Pingoru
Aug 08, 2024, 03:21 AM UTC
Resolved
Aug 09, 2024, 12:02 AM UTC
Duration
20h 40m
Affected: Block storage (nz-por-1)
Timeline · 2 updates
  1. investigating Aug 08, 2024, 03:21 AM UTC

    We are performing block storage maintenance in Porirua.

  2. resolved Aug 09, 2024, 12:02 AM UTC

    Block storage maintenance is now complete.

Read the full incident report →

Minor August 6, 2024

Billing dashboard and user management pages partially unaccessible

Detected by Pingoru
Aug 06, 2024, 11:21 PM UTC
Resolved
Aug 07, 2024, 12:45 AM UTC
Duration
1h 23m
Affected: Dashboard
Timeline · 3 updates
  1. investigating Aug 06, 2024, 11:21 PM UTC

    We are currently investigating this issue.

  2. monitoring Aug 06, 2024, 11:46 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Aug 07, 2024, 12:45 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor August 6, 2024

Block storage maintenance in Porirua

Detected by Pingoru
Aug 06, 2024, 09:10 PM UTC
Resolved
Aug 08, 2024, 03:19 AM UTC
Duration
1d 6h
Affected: Block storage (nz-por-1)
Timeline · 4 updates
  1. investigating Aug 06, 2024, 09:10 PM UTC

    We are performing maintenance activities on block storage in Porirua. There is no expected impact on workloads.

  2. investigating Aug 07, 2024, 11:04 PM UTC

    Some block storage volumes using the b1.standard storage tier may be experiencing slower read/write times than normal. We are currently investigating the issue.

  3. investigating Aug 08, 2024, 01:56 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Aug 08, 2024, 03:19 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor August 4, 2024

Unable to create new load balancers

Detected by Pingoru
Aug 04, 2024, 08:03 AM UTC
Resolved
Aug 04, 2024, 12:20 PM UTC
Duration
4h 16m
Affected: Load balancer (nz-por-1)
Timeline · 4 updates
  1. investigating Aug 04, 2024, 08:03 AM UTC

    We are currently investigating this issue.

  2. identified Aug 04, 2024, 08:04 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Aug 04, 2024, 09:02 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Aug 04, 2024, 12:20 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor August 4, 2024

Intermittent networking issues

Detected by Pingoru
Aug 04, 2024, 07:19 AM UTC
Resolved
Aug 04, 2024, 09:21 AM UTC
Duration
2h 2m
Affected: Network (nz-por-1)
Timeline · 3 updates
  1. investigating Aug 04, 2024, 07:19 AM UTC

    We are currently investigating this issue.

  2. identified Aug 04, 2024, 07:50 AM UTC

    The issue has been identified and a fix is being implemented.

  3. resolved Aug 04, 2024, 09:21 AM UTC

    This incident has been resolved.

Read the full incident report →

Notice July 29, 2024

Block Storage maintenance in Hamilton

Detected by Pingoru
Jul 29, 2024, 09:30 PM UTC
Resolved
Aug 01, 2024, 11:58 PM UTC
Duration
3d 2h
Affected: Block storage (nz-hlz-1)
Timeline · 2 updates
  1. investigating Jul 29, 2024, 09:30 PM UTC

    We are upgrading some block storage components in Hamilton this week. There is no expected user impact.

  2. resolved Aug 01, 2024, 11:58 PM UTC

    Block storage maintenance is now complete

Read the full incident report →

Minor July 22, 2024

Network Transit issues

Detected by Pingoru
Jul 22, 2024, 03:33 AM UTC
Resolved
Jul 22, 2024, 09:02 PM UTC
Duration
17h 28m
Affected: APIs (nz-hlz-1)APIs (nz-por-1)Object storage (nz-hlz-1)Object storage (nz-por-1)Network (nz-hlz-1)Network (nz-por-1)Load balancer (nz-hlz-1)Load balancer (nz-por-1)
Timeline · 3 updates
  1. investigating Jul 22, 2024, 03:33 AM UTC

    Our upstream ingress protection (cloudflare) is currently experiencing issues in New Zealand. Follow https://www.cloudflarestatus.com/incidents/7znjbdb5bdcd for more frequent updates.

  2. identified Jul 22, 2024, 03:33 AM UTC

    The issue has been identified and a fix is being implemented.

  3. resolved Jul 22, 2024, 09:02 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor June 26, 2024

The APIs in nz-por-1 are experiencing a degraded performance and are slow to serve requests

Detected by Pingoru
Jun 26, 2024, 01:10 AM UTC
Resolved
Jun 26, 2024, 05:30 AM UTC
Duration
4h 20m
Affected: APIs (nz-por-1)
Timeline · 6 updates
  1. investigating Jun 26, 2024, 01:10 AM UTC

    We are currently investigating this issue.

  2. investigating Jun 26, 2024, 01:37 AM UTC

    Customers may experience issues in launching, resizing or unshelving instances.

  3. identified Jun 26, 2024, 02:37 AM UTC

    We have identified the issue and are working on a solution.

  4. monitoring Jun 26, 2024, 04:52 AM UTC

    A fix has been implemented. Requests are likely to be slow for a while longer, we will continue monitoring and update status page once this period is over.

  5. monitoring Jun 26, 2024, 05:13 AM UTC

    Our fix was successful. Please re-try any actions that failed during this incident.

  6. resolved Jun 26, 2024, 05:30 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor June 25, 2024

Intermittent dashboard connectivity issues

Detected by Pingoru
Jun 25, 2024, 10:13 PM UTC
Resolved
Jun 25, 2024, 10:40 PM UTC
Duration
26m
Affected: Dashboard
Timeline · 2 updates
  1. investigating Jun 25, 2024, 10:13 PM UTC

    We are currently investigating this issue.

  2. resolved Jun 25, 2024, 10:40 PM UTC

    This incident has been resolved.

Read the full incident report →