Crusoe Outage History

We are investigating an issue on the eu-iceland1-storage cluster. Customers may experience elevated write latency. We will provide an update as soon as we have more information available.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Customers using CCR (Crusoe Container Registry) will be unable to push or pull images. We are working on resolving this matter as soon as possible. If you experience any other issues, please reach out to [email protected]

We are investigating an issue affecting access to Crusoe Managed Inference, including Intelligence Foundry and Bring Your Own Model (BYOM). Customers may be unable to reach inference endpoints or send requests to the service. Engineering is actively working on recovery. We will provide an update when more information is available.

We're investigating failures when creating new projects. Existing workloads and resources remain fully operational. We will provide an update when more information is available.

The issue has been mitigated and project creation requests are completing successfully.

We are currently experiencing an issue with storage used disk capacity metrics in our Iceland (eu-iceland1) region. All data access, read/write operations, and service functionality continue to operate normally with no performance impact. Our team is actively working to restore capacity metrics. We will provide updates as restoration progresses.

We are currently investigating an issue specifically affecting Persistent Volume (PVC) creation for Crusoe Managed Kubernetes (CMK) customers. While this interruption prevents the provisioning of new block storage volumes, it is strictly limited to block storage and does not impact Shared Disks or other services. Existing volumes and active workloads remain fully operational and unaffected. Our engineering team has identified the root cause and is restoring the service. We expect to provide a final update once provisioning functionality is fully restored.

We are investigating an issue in our eu-iceland1 region that may be impacting the ability to manage resources. Our engineering team is actively working to identify the scope and cause. We will provide another update as more information becomes available.

We are aware of an issue with an upstream dependency that may impact the ability to add new nodes to existing node pools. Existing clusters and running workloads are not affected. We are actively monitoring the situation and will provide updates as it progresses. If you are experiencing issues, please contact Crusoe Support.

We are investigating an issue affecting Kubernetes CMK control plane operations. Customers may experience elevated latency or timeouts when retrieving cluster credentials via the CLI or console, and may be unable to create new clusters. Existing cluster workloads are not affected. We will provide a further update as our investigation continues.

We are investigating an issue in our US-Southcentral region that may be impacting the ability to manage cluster resources via the API. Our engineering team is actively working to identify the scope and cause. We will provide another update as more information becomes available.

We are pleased to report that this incident is now resolved. We have confirmed that control plane functionality for all impacted clusters in the us-southcentral region has been fully restored. The issue was caused by network congestion on a small number of underlying physical hosts. This has been remediated by isolating the source of the traffic. We will be conducting a full post-mortem to prevent a recurrence and apologize for any disruption this incident caused.

We are currently investigating an issue impacting our Shared Disks in the eu-iceland1-a region. We are actively working with our storage vendor to resolve this issue. We will provide another update as soon as we have more information.

We are continuing to investigate this issue.

We are currently investigating an issue impacting our Shared Disks in the eu-iceland1-a region. Customers may not be able to create, read, update, and delete operations. VM operations may also be impacted. We are working with our Storage vendor on a resolution currently.

We are currently investigating an issue affecting control plane operations for shared storage in the eu-iceland1-a region. Existing shared volumes and running VMs are not affected, but creation and deletion of shared volumes may be impacted. We are actively working with our storage vendor to resolve this issue. We will provide another as soon as we have more information.

We are currently investigating reports of failures when creating Custom Images in our EU-Iceland (eu-iceland1-a) region. Users may receive "Internal Server Errors" or experience operation timeouts when attempting to create Custom Images from existing disks. If you experience any other issues, please reach out to [email protected].

Customers using CCR (Crusoe Container Registry ) will be unable to push or pull images in us-east1, eu-iceland1 and us-southcentral1. We are working on resolving this matter as soon as possible. If you experience any other issues, please reach out to [email protected]

We are currently investigating an issue that is causing some Direct NFS mounts to be unavailable in the eu-iceland1 region. Customers utilizing Direct NFS for shared drives may be unable to access their mounts. We are actively diagnosing the problem. We will post another update as soon as more information is available.

It appears that several storage backend VIPs are experiencing connectivity problems. We are actively collaborating with both our networking vendor and our storage vendor to identify and implement a resolution.

We are currently experiencing a network instability issue in eu-iceland1-a region, causing a subset of virtual machines to lose access to their boot disk. This is resulting in multiple VMs (Virtual Machines) throwing DNR (Device Not Ready) errors. We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.

We are currently investigating an issue affecting access to the Cloud Console UI and API. Users may be unable to log in to our cloud services or access the management dashboard.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

The issue is resolved and Cloud Console is accessible and functioning as expected.

We are currently investigating an incident affecting VM connectivity and performance in the Iceland region. Our Engineering teams are actively engaged and working to isolate the root cause. We will provide updates as our investigation progresses.

We have isolated the issue to a subset of VMs within the Iceland region and determined that it is not a core network transport failure. Our Engineering teams are continuing to actively troubleshoot the issue to restore service as soon as possible.

We are continuing to investigate this issue.

We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully. Existing, currently running VMs are operational. We advise customers to avoid rebooting critical workloads until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

We are continuing to investigate this issue.

We have determined that the problem is sporadic and limited to a specific subset of our host nodes. Existing, currently running VMs are operational. We still strongly recommend customers avoid rebooting critical workloads until the permanent fix is confirmed to be fully deployed. If you are currently experiencing issues with starting or restarting VMs, please open a support ticket so we can immediately address your specific host. We will provide another update once the targeted fix has been deployed to the identified hosts and we have confirmed stability.

We are continuing to investigate this issue.

We are continuing to investigate this issue.