pingoru pingoru Hyperstack had 42 outages in the last 2 years totaling 320h 27m of downtime, averaging 1.7 incidents per month.
There were 42 Hyperstack outages since May 1, 2025 totaling 320h 27m of downtime. Each is summarised below. Incident details, duration, and resolution information.
We are currently experiencing an electrical issue that might affect some systems, more information to come.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently experiencing a shortage of CPU-only nodes in the CA1 region. Customers may see errors when attempting to launch CPU-only instances. Our infrastructure team is actively working to bring additional capacity online to restore availability. GPU-backed instances are not affected.
This incident has been resolved.
We're experiencing intermittent electrical-related issues in NO1 region, a small selection of racks are currently identified as affected which in turn is causing unexpected reset of certain hosts. We're currently working with our on-site engineering team and are in the process of identifying an underlying cause and taking action.
This incident has been resolved.
There will be a short outage to AI Studio while backend services are reconfigured.
We are continuing to work on a fix for this issue.
This incident has been resolved.
At 1600 UTC we will be shutting down a storage node to perform maintenance. During this time there may be a slight decrease in IO performance for storage volumes.
We are continuing to work on a fix for this issue.
This incident has been resolved.
We are currently investigating a downed storage node. There may be a slight effect on performance as the cluster rebalances.
We have identified an inaccessible storage node and are attempting remediation to access.
The storage node has been brought back online and the cluster is now recovering. We will continue to monitor performance. While recovery is performed there may be a minor performance impact to storage in customer VMs.
This incident has been resolved.
We are currently investigating an outage affecting multiple nodes in the US1 region. We are currently working toward finding a resolution at this time.
We have been informed by our US1 data centre partner that a catastrophic power outage has impacted some pods and compute nodes, affecting a subset of customers. Our engineering team is working around the clock to bring new compute nodes online. For customers being affected, our Customer Support team will be reaching out directly to coordinate replacement capacity. Please note there may be temporary reduced stock availability in the US1 region while recovery efforts continue. We will provide further updates as more information becomes available.
We have successfully restored capacity in the US1 region following the recent outage at our data centre. Additional compute nodes have been brought online, and stock levels have been replenished to meet on-demand customer requirements. Our engineering and support teams will continue to monitor the environment closely to ensure stability. We thank you for your patience during the recovery process.
We're currently investigating a storage-related issue in our CA1 region. A subset of storage nodes is unavailable, which is affecting performance and availability for some virtual machines and operations that rely on block storage. - You may experience delays launching, hibernating, or snapshotting instances. - Instances booting from volume may be temporarily unavailable or exhibit degraded I/O performance. - General instance operation is unaffected if you are using root disk or ephemeral. Our engineering team is working with our hardware partner to restore full service as soon as possible. We’ll continue to provide updates here as more information becomes available.
Our CA1 data centre experienced a power disruption. While critical infrastructure responded as expected, some backend systems remain affected and we are actively working with our partners to restore full service.
All services are now restored in CA1. However, clients might still expected IOs degradation when our storage nodes are being rebalanced.
This incident has been resolved.
We are currently investigating a network connectivity issue affecting the NO1 region. Our team is working to identify the root cause and restore connectivity as quickly as possible. Updates will follow as more information becomes available.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently experiencing an API service issue affecting instance creation and network configuration changes. Our engineering team has identified the root cause and is actively working on remediation. Existing instances and network configurations remain unaffected.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
The fix was successfully tested and we are actively rolling it out in our Canada1 region. VM creation and management will be recovered by GPU types, starting with the RTX A6000.
The issue impacting instance creation and network configuration changes in the CA1 region has been resolved. As of now, all GPU types have been restored to normal operation, and VM creation and management are functioning as expected. We appreciate your patience while our engineering team worked to implement and verify the solution.
We are currently performing a storage expansion in our NO1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate your patience and understanding. If you encounter any issues, please contact our support team.
The expansion has finished. Thank you for your patience.
NexGen wishes to inform you that Montreal is currently experiencing an extreme heatwave, with temperatures reaching up to 46°C. This unprecedented heat is placing significant strain on our data centre infrastructure in CA-1 region, potentially affecting system stability. We are actively monitoring the situation and are implementing measures to minimize any impact on your services. We appreciate your understanding and continued support during this time.
This incident has been resolved.
We are currently experiencing a disruption with Hyperstack services in the US1 region. Existing workloads continue to run without interruption; however, API calls to this region are temporarily unavailable. Our team is investigating and will provide updates as they become available. We appreciate your patience.
This incident has been resolved.
We are currently addressing an issue within our CA1 region storage cluster. One of the storage nodes has experienced a failure and is being investigated. While the system is designed to self-heal and redistribute data automatically, you may notice temporary performance degradation during this process. Our engineering team is actively monitoring the situation to ensure a swift and smooth recovery. We will provide updates as the recovery progresses.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
CA1 region experienced a complete network outage lasting approximately 5 minutes due to an upstream issue with our internet service provider. The disruption impacted all external connectivity, including access to public services and remote management. Internal services remained operational. While connectivity has since been restored, the region is currently experiencing degraded network performance. Users may notice increased latency and reduced bandwidth for inbound and outbound connections. We are actively monitoring the situation and working to resolve the issue and restore full performance.
We are observing ongoing degraded performance on a small number of routes. Most network traffic remains stable and unaffected. We are continuing to monitor the situation closely and are working with our upstream provider to ensure full resolution.
This incident has been resolved.
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate your patience and understanding. If you encounter any issues, please contact our support team.
A fix has been implemented and we are monitoring the results.
The storage expansion in CA1 has finished. Thank you for your understanding.
We are currently investigating a service outage affecting the US-1 Data Center. Please note that this incident does not impact any running workloads, but you may experience issues when launching new services. We sincerely apologise for the inconvenience and will provide further updates as we learn more. Thank you for your patience.
This incident has been resolved.