Crusoe Outage History

We are currently investigating an issue impacting virtual machine accessibility in eu-iceland1-a region. Some users may be unable to access their VMs at this time. We appreciate your patience and will provide updates as more information becomes available.

We're making significant progress in resolving the virtual machine accessibility issues in the eu-iceland1-a region. We apologize for the continued disruption and thank you for your patience as we work to restore full service. We will provide another update as soon as more information is available or when services are fully restored. You can continue to contact our support team at [email protected], if there are any other concerns.

We are currently investigating an issue with machine accessibility and performance issues in the us-southcentral1 region . Some users may be unable to access their VMs at this time or see performance degradation. We appreciate your patience and will provide updates as more information becomes available.

Please be aware that the user interface is currently not showing the correct capacity for CMK nodepool creation, even when quota is available. This is a UI-specific bug. As a temporary workaround, please use the command-line interface (CLI) to successfully create nodepools. Our team is investigating the root cause and will deploy a fix as soon as possible.

We are currently investigating an issue impacting the ability to start Virtual Machines (VMs) in our us-southcentral1-a region. Our engineering team has identified a potential issue and is actively working on a resolution. Thank you for your patience as we work to restore full functionality.

We're investigating a service degradation in our us-east1-a region, triggered by a facility power disruption at our data center. The primary impact is to the Infiniband networking fabric, which may cause intermittent errors or failures for multi-node, distributed workloads. Some customers may also experience individual virtual machines becoming unavailable. Our teams are working to identify all affected resources. Our engineering teams are actively working to stabilize the affected systems and mitigate the risk of further disruption. We are coordinating with our data center provider to support their remediation efforts and restore full service resiliency as quickly as possible. We apologize for any impact this is causing.

We are continuing to investigate and mitigate the service degradation affecting our us-east1-a region, following a facility power disruption at our data center. Our teams remain in close coordination with the data center provider as they work to fully restore services. Recovery of critical systems remains our top priority. We sincerely apologize for the ongoing impact and appreciate your continued patience as we work to resolve the issue.

We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

We are currently investigating an issue affecting VM's in eu-iceland1

We're currently investigating an issue impacting our control plane in the US-Southcentral region. Customers may experience errors or timeouts when making certain API calls, including those for creating new virtual machines. Our engineering team is investigating the underlying cause with the highest priority. We appreciate your patience and will provide another update as soon as we have more information.

We are currently investigating a potential service interruption that may impact VMs to be unreachable in the us-northcentral1-a region. If you are experiencing issues in this region, please reach out to [email protected]

Our teams are working diligently and believe we have identified the root cause and we are working on implementing a resolution. We will continue to provide updates on this page as further developments occur.

We are continuing our in-depth investigation into the intermittent storage connectivity issues impacting VM reachability in the us-northcentral1-a region. We will provide more information as it becomes available.

Our team is continuing its in-depth investigation into the intermittent storage connectivity issues impacting VM reachability in the us-northcentral1-a region, in collaboration with our vendor. We will keep you posted on further developments

We are continuing our dedicated investigation into the intermittent storage connectivity issues impacting VM reachability in us-northcentral1-a. Our teams are working in close collaboration with our storage vendor to analyze the underlying cause. We recognize the impact this is having and have all necessary resources allocated to resolving this.

We are seeing signs of recovery, and some previously impacted VMs in the us-northcentral1-a region are now available. Our teams are closely monitoring the environment and continuing their work with our vendor to restore full service for all remaining affected instances.

We are currently experiencing an outage with our 3rd party service provider, which is intermittently impacting the ability to provision and manage Virtual Machines (VMs) across all of our regions.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are expanding the scope of this incident to include the Crusoe console being inaccessible. This means that users are currently unable to log in, view, or manage resources through the Crusoe console interface. We are investigating this alongside the existing issues.

We are currently investigating an issue impacting virtual machine accessibility. Some users may be unable to stop and start Virtual Machines.