Crusoe Outage History

Started Feb 27, 2026, 04:58 AM UTC· resolved Mar 04, 2026, 09:05 PM UTC

1. investigating Feb 27, 2026, 04:58 AM UTC

   We are currently investigating an issue impacting our Shared Disks in the eu-iceland1-a region. We are actively working with our storage vendor to resolve this issue. We will provide another update as soon as we have more information.

2. investigating Feb 27, 2026, 04:58 AM UTC

   We are continuing to investigate this issue.

3. monitoring Feb 27, 2026, 05:36 AM UTC

   The issue has been mitigated and is now operating normally. We are monitoring the system to ensure continued stability.

4. resolved Mar 04, 2026, 09:05 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 20, 2026, 12:59 PM UTC· resolved Feb 20, 2026, 03:21 PM UTC

1. investigating Feb 20, 2026, 12:59 PM UTC

   We are currently investigating an issue impacting our Shared Disks in the eu-iceland1-a region. Customers may not be able to create, read, update, and delete operations. VM operations may also be impacted. We are working with our Storage vendor on a resolution currently.

2. resolved Feb 20, 2026, 03:21 PM UTC

   A fix has been implemented which fixed the issue. All operations should have resumed normally.

Read the full incident report →

Started Feb 04, 2026, 12:19 AM UTC· resolved Feb 04, 2026, 05:07 AM UTC

1. investigating Feb 04, 2026, 12:19 AM UTC

   We are currently investigating an issue affecting control plane operations for shared storage in the eu-iceland1-a region. Existing shared volumes and running VMs are not affected, but creation and deletion of shared volumes may be impacted. We are actively working with our storage vendor to resolve this issue. We will provide another as soon as we have more information.

2. monitoring Feb 04, 2026, 12:50 AM UTC

   Our storage vendor has successfully mitigated the issue. Control plane operations are now fully operational. We will monitor for some time before declaring the issue resolved.

3. resolved Feb 04, 2026, 05:07 AM UTC

   This issue has been resolved. If you are still experiencing any other issues related to this matter, please contact [email protected].

Read the full incident report →

Started Feb 03, 2026, 09:42 PM UTC· resolved Feb 04, 2026, 12:12 AM UTC

1. investigating Feb 03, 2026, 09:42 PM UTC

   We are currently investigating reports of failures when creating Custom Images in our EU-Iceland (eu-iceland1-a) region. Users may receive "Internal Server Errors" or experience operation timeouts when attempting to create Custom Images from existing disks. If you experience any other issues, please reach out to [email protected].

2. resolved Feb 04, 2026, 12:12 AM UTC

   This issue has been resolved. If you are still experiencing any other issues related to this matter, please contact [email protected].

Read the full incident report →

Started Feb 01, 2026, 09:42 PM UTC· resolved Feb 02, 2026, 04:38 AM UTC

1. investigating Feb 01, 2026, 09:42 PM UTC

   Customers using CCR (Crusoe Container Registry ) will be unable to push or pull images in us-east1, eu-iceland1 and us-southcentral1. We are working on resolving this matter as soon as possible. If you experience any other issues, please reach out to [email protected]

2. resolved Feb 02, 2026, 04:38 AM UTC

   This issue has been resolved. Please contact [email protected] if you are still experiencing any issues.

Read the full incident report →

Started Dec 29, 2025, 09:28 AM UTC· resolved Jan 02, 2026, 08:47 PM UTC

1. investigating Dec 29, 2025, 09:28 AM UTC

   We are currently investigating an issue that is causing some Direct NFS mounts to be unavailable in the eu-iceland1 region. Customers utilizing Direct NFS for shared drives may be unable to access their mounts. We are actively diagnosing the problem. We will post another update as soon as more information is available.

2. investigating Dec 29, 2025, 02:00 PM UTC

   It appears that several storage backend VIPs are experiencing connectivity problems. We are actively collaborating with both our networking vendor and our storage vendor to identify and implement a resolution.

3. monitoring Dec 29, 2025, 10:29 PM UTC

   We have restored connectivity to all affected Direct NFS mounts by applying a mitigation to the network hardware. Service is now operating normally and we are monitoring the system to ensure continued stability.

4. resolved Jan 02, 2026, 08:47 PM UTC

   This issue is resolved and connectivity to all Direct NFS mounts in the eu-iceland1 region has been fully restored. The network mitigation is verified, and the system is stable.

Read the full incident report →

Started Dec 16, 2025, 06:19 PM UTC· resolved Dec 17, 2025, 09:12 AM UTC

1. investigating Dec 16, 2025, 06:19 PM UTC

   We are currently experiencing a network instability issue in eu-iceland1-a region, causing a subset of virtual machines to lose access to their boot disk. This is resulting in multiple VMs (Virtual Machines) throwing DNR (Device Not Ready) errors. We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.

2. resolved Dec 17, 2025, 09:12 AM UTC

   A fix has been applied and the issue regarding "Device Not Ready" (DNR) errors in the eu-iceland1-a region is now resolved. All impacted services and Virtual Machines have returned to normal operations.

Read the full incident report →

Started Dec 16, 2025, 05:13 PM UTC· resolved Jan 07, 2026, 05:25 PM UTC

1. investigating Dec 16, 2025, 05:13 PM UTC

   We are currently investigating slowness when using Shared Storage in eu-iceland1-a. We are working with our storage vendor to resolve the issue as soon as possible.

2. investigating Dec 16, 2025, 05:36 PM UTC

   We are expanding the scope of impact for this incident. Customers will observe significant slowness and intermittent access issues when utilizing Shared Storage services in eu-iceland1-a region. We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.

3. investigating Dec 16, 2025, 06:26 PM UTC

   We have implemented a fix for the storage performance issues in eu-iceland1-a. Storage latency and throughput have returned to normal operating levels. We are actively monitoring the service to ensure continued stability.

4. investigating Dec 16, 2025, 06:53 PM UTC

   We are continuing to observe impact related to the storage performance degradation in eu-iceland1-a. Our engineering teams remain actively engaged with our vendor in focused remediation efforts and are working to address residual intermittent slowness for some customers utilizing Shared Storage services. We will provide another update as soon as we have further developments. Thank you for your continued patience

5. investigating Dec 16, 2025, 10:43 PM UTC

   Engineering teams are continuing to actively remediate specific storage metadata inconsistencies and stabilize network connectivity within the eu-iceland1-a region. During this window, customers may continue to experience intermittent write errors, latency spikes, or brief connectivity drops. We are prioritizing work to restore full service performance and expect to provide further details within approximately 2 hours.

6. identified Dec 17, 2025, 12:49 AM UTC

   Engineering teams are currently executing a targeted integrity verification process to permanently resolve the metadata inconsistencies reported earlier. To ensure strict data safety during this operation, the storage cluster has entered a protective maintenance state that temporarily restricts access to prevent modification while checks are running. As a result of these active safety measures, customers in eu-iceland1-a may experience service unavailability, including mount failures or Input/Output (I/O) errors, for specific volumes, directories, or files. Please be assured that these restrictions are intentional precautions to preserve data integrity. We expect to lift these protective measures once the verification process is complete.

7. monitoring Dec 17, 2025, 02:05 AM UTC

   Engineering teams have completed the critical phase of the integrity verification process and have lifted the broad protective maintenance restrictions. As a result, service availability has been restored for the majority of volumes and data paths in the eu-iceland1-a region. However, a small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors. We have transitioned to a monitoring phase and teams are actively working to restore full access to these remaining items. We will provide a final confirmation once the incident is fully resolved.

8. identified Dec 17, 2025, 02:53 AM UTC

   Hello Team, during our monitoring we are observing VM operation failures, as well as shared volume create/update/delete failures due to ongoing impact to the storage control plane. Some customers may still observe issues when accessing existing volumes. A small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors. We are actively engaged with our vendor and pursuing further investigation on this matter.

9. investigating Dec 17, 2025, 09:22 PM UTC

   We are continuing to address the ongoing service instability affecting shared volumes in the eu-iceland1-a region. While engineering teams continue the previously reported background recovery efforts, we are observing intermittent performance fluctuations and connectivity drops, which may result in transient errors for active workloads. We are actively engaged with our storage vendor to diagnose the specific drivers of this instability and are working to restore consistent performance to the cluster.

10. identified Dec 17, 2025, 11:11 PM UTC

    Engineering teams have implemented configuration changes to isolate background recovery processes from active customer workloads. Following these adjustments, we are observing signs of stabilization in cluster performance and a reduction in the intermittent connectivity fluctuations reported earlier. We are continuing to monitor service health closely. Please note that while stability has improved, further maintenance operations may be required to finalize the recovery efforts. We are currently evaluating the necessity and timing of these potential steps with the vendor and will provide updates as plans are finalized.

11. identified Dec 18, 2025, 06:37 PM UTC

    The majority of customers should now observe a return to normal operations and consistent performance within the eu-iceland1-a region. While general cluster stability has been restored, engineering teams are continuing focused recovery efforts for a limited subset of customers who remain more deeply impacted by residual metadata inconsistencies. We are maintaining close oversight of service health and will continue to work directly with our storage vendor to finalize remediation for the remaining impacted volumes. Further updates will be provided as the recovery process nears full completion

12. identified Dec 18, 2025, 11:00 PM UTC

    The storage cluster continues to operate with stable performance for the majority of workloads. Regarding the subset of volumes requiring further intervention, we are closely coordinating with our storage vendor to execute comprehensive integrity scans and fully map the scope of residual inconsistencies. In parallel, the vendor is currently finalizing the development of automated remediation workflows, modeled on successful manual recovery procedures validated on a limited data subset earlier today. We are closely monitoring this development and will only authorize the execution of these workflows once the solution has been rigorously validated. We will provide further details as we prepare to transition from these ongoing scans to the active recovery phase.

13. identified Dec 23, 2025, 06:41 AM UTC

    To accelerate the VAST recovery progress at scale, we need to deploy an upgrade to the underlying VAST storage software. We are planning a maintenance window at 12:00 am PT on December 23rd. Target Start Time: 12:00 AM PST Expected Duration: 4 Hours Purpose: This VAST upgrade introduces new automated tooling designed to accelerate recovery workflows. Customer Impact: We do not anticipate any downtime; however, you may experience some service degradation (e.g. temporary latency or brief I/O pauses). If you have specific concerns or questions regarding this maintenance window, please reach out to [email protected]

14. identified Dec 23, 2025, 12:00 PM UTC

    The planned VAST storage software upgrade has been completed successfully across the cluster. We are now transitioning back to active recovery operations for the remaining volumes requiring remediation. With the new automated tooling now in place, engineering teams are resuming the execution of the validated remediation workflows. We will continue to monitor the progress and stability of the cluster as these recovery efforts scale.

15. identified Dec 23, 2025, 08:21 PM UTC

    We observed a blip that affected our storage cluster in the eu-iceland1-a region from 2025-12-23 19:52:53 UTC to 2025-12-23 20:03:47 UTC. This would have resulted in read, write, and update operations failing on all shared volumes. The service should be restored now. Our engineering team is working with our storage vendor to investigate the cause of this.

16. identified Dec 23, 2025, 09:22 PM UTC

    We have completed our initial review of an approximately 10 minute storage outage in eu-iceland1-a region. The interruption was caused by a manual administrative error by our storage provider (VAST) during the data recovery process. While service was restored within minutes, we recognize that any downtime is unacceptable. Our storage partner, VAST, is conducting a full post incident review and implementing additional automation and safeguards to eliminate reliance on manual steps and reduce the risk of similar issues in the future.

17. identified Dec 26, 2025, 11:17 AM UTC

    We have identified a new storage network issue impacting accessibility of shared volumes in the eu-iceland1-a region. We are currently engaged with our storage partner, VAST, to remediate the issue and restore stable access. Further updates will be provided as we make progress.

18. identified Dec 26, 2025, 02:39 PM UTC

    The storage network issues in the eu-iceland1-a region have been resolved, and access to shared volumes is restored. Together with our vendor VAST, we have stabilized the network layer and verified that I/O operations are performing normally. We will continue to monitor the environment closely to ensure continued stability.

19. identified Dec 26, 2025, 10:54 PM UTC

    We have identified a new storage network issue impacting accessibility of shared volumes in the eu-iceland1-a region. We are currently engaged with our storage partner, VAST, to remediate the issue and restore stable access. Further updates will be provided as we make progress.

20. identified Dec 26, 2025, 11:40 PM UTC

    We are expanding the scope of impact for the ongoing storage network issue in the eu-iceland1-a region. In addition to the accessibility of shared volumes, this issue is now also impacting Virtual Machine (VM) operations, including create, start, stop, and delete. We are actively working with our storage partner, VAST, to remediate the problem. They are in the process of rebooting the cluster to restore access. Further updates will be provided as we make progress.

21. identified Dec 27, 2025, 03:45 AM UTC

    We have confirmed that the storage network issue in the eu-iceland1-a region is now resolved. The storage cluster is currently stable. Access to shared volumes should be restored for everyone, and all Virtual Machine (VM) operations—including create, start, stop, and delete—have returned to normal functionality. We are monitoring the environment closely alongside our partner, VAST, to ensure ongoing stability and will keep you posted on any further developments or changes to this status.

22. identified Dec 27, 2025, 05:23 PM UTC

    Our storage vendor will be applying a configuration update to the underlying storage nodes starting at 10:00 AM PT, which will take approximately 15 minutes to complete. This maintenance is designed to optimize and increase the speed of existing recovery workflows. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please reach out to [email protected].

23. identified Dec 27, 2025, 06:23 PM UTC

    The planned VAST configuration update has been completed successfully across the cluster. We are now transitioning back to active recovery operations for the remaining volumes requiring remediation. We will continue to monitor the progress and stability of the cluster as these recovery efforts continue.

24. identified Dec 28, 2025, 01:03 AM UTC

    We are upgrading the VAST storage software at 6:00 AM PT on December 28th to further optimize recovery progress. This maintenance window is required to deploy critical improvements to the recovery process and has an expected duration of 4 hours. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please reach out to [email protected].

25. identified Dec 28, 2025, 11:10 AM UTC

    We have identified an issue with the underlying storage infrastructure in the eu-iceland1-a region. Several storage components have gone offline unexpectedly, impacting the availability of shared volumes and preventing VMs management operations. We are working urgently with our vendor, VAST, to restore these components and recover service. We understand the impact this has on your active workloads and are treating this with the highest priority. This is unrelated to the VAST upgrade planned at 6:00 AM PT on December 28th, which hasn't started yet.

26. identified Dec 28, 2025, 01:48 PM UTC

    We have restored full service to the eu-iceland1-a storage cluster. The storage node failures have been remediated, and shared disks are once again performing normally. VMs operations have also been stabilized. We sincerely apologize for the continued disruption. Our teams are now working with the vendor on an exhaustive review of the cluster's recent stability to prevent future recurrences. VAST upgrade at 6:00 AM PT will proceed as planned.

27. identified Dec 28, 2025, 03:18 PM UTC

    To ensure the reliability of the system after today's issue, we are postponing the VAST storage software upgrade, initially planned at 6:00 AM PT on December 28th, to 9:00 AM PT. This maintenance window is required to deploy critical improvements to the recovery process and has an expected duration of 4 hours. The new upgrade window is 9:00 PT to 13:00 PT. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please reach out to [email protected].

28. identified Dec 28, 2025, 09:53 PM UTC

    The planned VAST storage software upgrade, which began at 9:00 AM PT, has been completed successfully. The storage cluster is stable, and all services are operating normally.

29. identified Dec 30, 2025, 07:23 PM UTC

    We are upgrading the VAST storage software at 12:00 PM PT today, December 30th, to optimize recovery progress further. This maintenance window is required to deploy critical improvements to the recovery process and has an expected duration of 4 hours. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please contact [email protected].

30. identified Dec 31, 2025, 12:45 AM UTC

    The planned VAST storage software upgrade, which began at 12:00 PM PT, completed successfully. The storage cluster is stable, and all services are operating normally.

31. identified Dec 31, 2025, 03:45 PM UTC

    We are performing an additional upgrade to the VAST storage software today, December 31st, at 9:00 AM PT, to further optimize and accelerate the recovery process. While we do not anticipate any downtime, you may experience temporary service degradation during this period. We appreciate your continued patience as we work through these final stages of recovery. If you have specific concerns or questions about this maintenance window, please don't hesitate to contact us at [email protected].

32. identified Dec 31, 2025, 08:53 PM UTC

    The planned VAST storage software upgrade, which began at 9:00 AM PT, completed successfully. The storage cluster is stable, and all services are operating normally.

33. identified Jan 01, 2026, 05:01 PM UTC

    VAST is performing an additional upgrade to the storage software in our eu-iceland1-a region today, January 1st, at 10:00 AM PT. While we do not anticipate any downtime, you may experience temporary service degradation during this period. We appreciate your continued patience as we work through these final stages of recovery. If you have specific concerns or questions about this maintenance window, please don't hesitate to contact us at [email protected].

34. identified Jan 02, 2026, 06:06 PM UTC

    As of this morning, all planned activities to restore storage access have been completed. As we transition the system back to normal operations, we are now moving into a monitoring phase.

35. monitoring Jan 02, 2026, 06:47 PM UTC

    As of this morning, all planned activities to restore storage access have been completed. As we transition the system back to normal operations, we are now moving into a monitoring phase.

36. resolved Jan 07, 2026, 05:25 PM UTC

    Final monitoring of the eu-iceland1-a storage cluster has been successfully completed, and all services remain stable. All planned recovery activities and software upgrades are finished, with performance and VM operations restored to normal levels. This incident is now resolved.

Read the full incident report →

Started Nov 18, 2025, 12:48 PM UTC· resolved Nov 18, 2025, 04:38 PM UTC

1. investigating Nov 18, 2025, 12:48 PM UTC

   We are currently investigating an issue affecting access to the Cloud Console UI and API. Users may be unable to log in to our cloud services or access the management dashboard.

2. investigating Nov 18, 2025, 02:28 PM UTC

   We are continuing to investigate this issue.

3. investigating Nov 18, 2025, 02:28 PM UTC

   We are continuing to investigate this issue.

4. investigating Nov 18, 2025, 04:37 PM UTC

   The issue is resolved and Cloud Console is accessible and functioning as expected.

5. resolved Nov 18, 2025, 04:38 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 30, 2025, 12:46 AM UTC· resolved Nov 04, 2025, 07:46 PM UTC

1. investigating Oct 30, 2025, 12:46 AM UTC

   We are currently investigating an incident affecting VM connectivity and performance in the Iceland region. Our Engineering teams are actively engaged and working to isolate the root cause. We will provide updates as our investigation progresses.

2. investigating Oct 30, 2025, 02:35 AM UTC

   We have isolated the issue to a subset of VMs within the Iceland region and determined that it is not a core network transport failure. Our Engineering teams are continuing to actively troubleshoot the issue to restore service as soon as possible.

3. investigating Oct 30, 2025, 02:36 AM UTC

   We are continuing to investigate this issue.

4. identified Oct 30, 2025, 04:55 AM UTC

   We have identified the issue and are deploying a fix.

5. monitoring Oct 30, 2025, 06:10 AM UTC

   We have mitigated the issue and connectivity is confirmed to be restored. We will monitor overnight.

6. resolved Nov 04, 2025, 07:46 PM UTC

   Issue resolved.

Read the full incident report →

Started Oct 09, 2025, 09:37 PM UTC· resolved Oct 14, 2025, 11:43 PM UTC

1. investigating Oct 09, 2025, 09:37 PM UTC

   We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully. Existing, currently running VMs are operational. We advise customers to avoid rebooting critical workloads until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

2. investigating Oct 09, 2025, 09:38 PM UTC

   We are continuing to investigate this issue.

3. investigating Oct 10, 2025, 01:59 AM UTC

   We have determined that the problem is sporadic and limited to a specific subset of our host nodes. Existing, currently running VMs are operational. We still strongly recommend customers avoid rebooting critical workloads until the permanent fix is confirmed to be fully deployed. If you are currently experiencing issues with starting or restarting VMs, please open a support ticket so we can immediately address your specific host. We will provide another update once the targeted fix has been deployed to the identified hosts and we have confirmed stability.

4. investigating Oct 10, 2025, 02:02 AM UTC

   We are continuing to investigate this issue.

5. investigating Oct 14, 2025, 11:42 PM UTC

   We are continuing to investigate this issue.

6. resolved Oct 14, 2025, 11:43 PM UTC

   Update: The issue is now resolved. The engineering team has implemented a fix and confirmed that normal provisioning operations have been restored.

Read the full incident report →

Started Oct 02, 2025, 04:00 AM UTC· resolved Oct 02, 2025, 08:50 PM UTC

1. identified Oct 02, 2025, 04:00 AM UTC

   We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

2. monitoring Oct 02, 2025, 08:20 PM UTC

   A fix has been implemented, and we are monitoring the environment.

3. resolved Oct 02, 2025, 08:50 PM UTC

   This incident is now resolved.

Read the full incident report →

Started Sep 30, 2025, 06:45 PM UTC· resolved Oct 01, 2025, 12:00 AM UTC

1. investigating Sep 30, 2025, 06:45 PM UTC

   We are currently investigating an issue impacting virtual machine accessibility in eu-iceland1-a region. Some users may be unable to access their VMs at this time. We appreciate your patience and will provide updates as more information becomes available.

2. investigating Sep 30, 2025, 07:48 PM UTC

   We're making significant progress in resolving the virtual machine accessibility issues in the eu-iceland1-a region. We apologize for the continued disruption and thank you for your patience as we work to restore full service. We will provide another update as soon as more information is available or when services are fully restored. You can continue to contact our support team at [email protected], if there are any other concerns.

3. identified Sep 30, 2025, 08:34 PM UTC

   We have identified and mitigated the issue. Although no further impact is expected, we are continuing to monitor. If you are still experiencing issues, please reach out to us through [email protected] and we will investigate further.

4. monitoring Sep 30, 2025, 09:55 PM UTC

   A fix has been applied and the outage has been mitigated. Services have returned to normal operations in eu-iceland1-a region. We are continuing to monitor. Please contact [email protected] if you are still experiencing any issues.

5. resolved Oct 01, 2025, 12:00 AM UTC

   This incident has been resolved. Please contact [email protected] if you are still experiencing any issues.

Read the full incident report →

Started Sep 30, 2025, 10:22 AM UTC· resolved Sep 30, 2025, 10:49 AM UTC

1. investigating Sep 30, 2025, 10:22 AM UTC

   We are currently investigating an issue with machine accessibility and performance issues in the us-southcentral1 region . Some users may be unable to access their VMs at this time or see performance degradation. We appreciate your patience and will provide updates as more information becomes available.

2. resolved Sep 30, 2025, 10:49 AM UTC

   A fix has been implemented. This incident is now resolved

Read the full incident report →

Started Sep 19, 2025, 09:53 PM UTC· resolved Sep 20, 2025, 01:14 AM UTC

1. investigating Sep 19, 2025, 09:53 PM UTC

   Please be aware that the user interface is currently not showing the correct capacity for CMK nodepool creation, even when quota is available. This is a UI-specific bug. As a temporary workaround, please use the command-line interface (CLI) to successfully create nodepools. Our team is investigating the root cause and will deploy a fix as soon as possible.

2. resolved Sep 20, 2025, 01:14 AM UTC

   The fix has been deployed, and the user interface now correctly displays the capacity for CMK nodepool creation. You can now use either the UI or the CLI to provision new nodepools.

Read the full incident report →

Started Sep 19, 2025, 07:33 AM UTC· resolved Sep 19, 2025, 01:20 PM UTC

1. investigating Sep 19, 2025, 07:33 AM UTC

   We are currently investigating an issue impacting the ability to start Virtual Machines (VMs) in our us-southcentral1-a region. Our engineering team has identified a potential issue and is actively working on a resolution. Thank you for your patience as we work to restore full functionality.

2. resolved Sep 19, 2025, 01:20 PM UTC

   A fix has been implemented. This incident is now resolved

Read the full incident report →

Started Aug 20, 2025, 05:09 PM UTC· resolved Aug 21, 2025, 11:35 PM UTC

1. investigating Aug 20, 2025, 05:09 PM UTC

   We're investigating a service degradation in our us-east1-a region, triggered by a facility power disruption at our data center. The primary impact is to the Infiniband networking fabric, which may cause intermittent errors or failures for multi-node, distributed workloads. Some customers may also experience individual virtual machines becoming unavailable. Our teams are working to identify all affected resources. Our engineering teams are actively working to stabilize the affected systems and mitigate the risk of further disruption. We are coordinating with our data center provider to support their remediation efforts and restore full service resiliency as quickly as possible. We apologize for any impact this is causing.

2. investigating Aug 21, 2025, 02:32 AM UTC

   We are continuing to investigate and mitigate the service degradation affecting our us-east1-a region, following a facility power disruption at our data center. Our teams remain in close coordination with the data center provider as they work to fully restore services. Recovery of critical systems remains our top priority. We sincerely apologize for the ongoing impact and appreciate your continued patience as we work to resolve the issue.

3. monitoring Aug 21, 2025, 07:59 PM UTC

   We have successfully mitigated the issue affecting the us-east1-a region. The facility power disruption has been addressed, and the impacted Infiniband networking fabric and associated systems have returned to normal operation. All affected services are now stable, and full functionality has been restored. Our teams will continue to monitor the region closely to ensure continued stability. We appreciate your patience during this incident and apologize again for any disruption it may have caused.

4. resolved Aug 21, 2025, 11:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 01, 2025, 03:18 AM UTC· resolved Aug 01, 2025, 06:21 AM UTC

1. investigating Aug 01, 2025, 03:18 AM UTC

   We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

2. identified Aug 01, 2025, 03:44 AM UTC

   The issue has been identified, and we have tested a fix internally. We are working on rolling out the fix to our A100 Infiniband type servers now.

3. monitoring Aug 01, 2025, 05:06 AM UTC

   A fix has been implemented, and we are monitoring the environment for now.

4. resolved Aug 01, 2025, 06:21 AM UTC

   This incident is now resolved

Read the full incident report →

Started Jul 24, 2025, 08:30 AM UTC· resolved Jul 24, 2025, 11:36 AM UTC

1. investigating Jul 24, 2025, 08:30 AM UTC

   We are currently investigating an issue affecting VM's in eu-iceland1

2. identified Jul 24, 2025, 10:26 AM UTC

   We have identified an issue with our persistent disk storage system for a subset of VM's in eu-iceland1. Our engineers are now working to restore service.

3. monitoring Jul 24, 2025, 10:52 AM UTC

   A fix has been implemented for the persistent disk storage system. We are observing recovery, and affected virtual machines in eu-iceland1 are returning to a healthy state. Our team will continue to monitor the platform to ensure full service restoration and stability. We will provide the next update once the incident is fully resolved. We appreciate your understanding and support. If you experience any issues, please don't hesitate to contact us at [email protected].

4. resolved Jul 24, 2025, 11:36 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 11, 2025, 02:28 PM UTC· resolved Jul 11, 2025, 04:21 PM UTC

1. investigating Jul 11, 2025, 02:28 PM UTC

   We're currently investigating an issue impacting our control plane in the US-Southcentral region. Customers may experience errors or timeouts when making certain API calls, including those for creating new virtual machines. Our engineering team is investigating the underlying cause with the highest priority. We appreciate your patience and will provide another update as soon as we have more information.

2. identified Jul 11, 2025, 03:11 PM UTC

   We've identified a potential issue on the control plane network and we're working to address the issue currently.

3. monitoring Jul 11, 2025, 03:32 PM UTC

   A fix has been applied and the outage has been mitigated. Services have returned to normal operations in US-Southcentral region. We are continuing to monitor the situation and validate preventive measures. Please contact [email protected] if you are still experiencing any issues.

4. resolved Jul 11, 2025, 04:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 30, 2025, 06:05 PM UTC· resolved Jul 02, 2025, 05:21 PM UTC

1. investigating Jun 30, 2025, 06:05 PM UTC

   We are currently investigating a potential service interruption that may impact VMs to be unreachable in the us-northcentral1-a region. If you are experiencing issues in this region, please reach out to [email protected]

2. investigating Jun 30, 2025, 10:01 PM UTC

   Our teams are working diligently and believe we have identified the root cause and we are working on implementing a resolution. We will continue to provide updates on this page as further developments occur.

3. investigating Jul 01, 2025, 01:34 AM UTC

   We are continuing our in-depth investigation into the intermittent storage connectivity issues impacting VM reachability in the us-northcentral1-a region. We will provide more information as it becomes available.

4. investigating Jul 01, 2025, 05:21 AM UTC

   Our team is continuing its in-depth investigation into the intermittent storage connectivity issues impacting VM reachability in the us-northcentral1-a region, in collaboration with our vendor. We will keep you posted on further developments

5. investigating Jul 01, 2025, 10:06 AM UTC

   We are continuing our dedicated investigation into the intermittent storage connectivity issues impacting VM reachability in us-northcentral1-a. Our teams are working in close collaboration with our storage vendor to analyze the underlying cause. We recognize the impact this is having and have all necessary resources allocated to resolving this.

6. investigating Jul 01, 2025, 02:14 PM UTC

   We are seeing signs of recovery, and some previously impacted VMs in the us-northcentral1-a region are now available. Our teams are closely monitoring the environment and continuing their work with our vendor to restore full service for all remaining affected instances.

7. monitoring Jul 01, 2025, 08:57 PM UTC

   A fix has been applied and the outage has been mitigated. Services have returned to normal operations in us-northcentral1-a region. We are continuing to monitor the situation and validate preventive measures. Please contact [email protected] if you are still experiencing any issues.

8. resolved Jul 02, 2025, 05:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 12, 2025, 07:55 PM UTC· resolved Jun 12, 2025, 09:35 PM UTC

1. investigating Jun 12, 2025, 07:55 PM UTC

   We are currently experiencing an outage with our 3rd party service provider, which is intermittently impacting the ability to provision and manage Virtual Machines (VMs) across all of our regions.

2. investigating Jun 12, 2025, 08:03 PM UTC

   We are continuing to investigate this issue.

3. investigating Jun 12, 2025, 08:06 PM UTC

   We are continuing to investigate this issue.

4. investigating Jun 12, 2025, 08:16 PM UTC

   We are expanding the scope of this incident to include the Crusoe console being inaccessible. This means that users are currently unable to log in, view, or manage resources through the Crusoe console interface. We are investigating this alongside the existing issues.

5. monitoring Jun 12, 2025, 09:08 PM UTC

   We're starting to see signs of recovery across affected services, and our team is monitoring the situation very closely.

6. resolved Jun 12, 2025, 09:35 PM UTC

   All Crusoe Cloud services have been restored and are fully operational. We will continue to monitor our platform. If you notice any issues please reach out to [email protected]

Read the full incident report →

Started Jun 11, 2025, 03:10 AM UTC· resolved Jun 11, 2025, 03:36 AM UTC

1. investigating Jun 11, 2025, 03:10 AM UTC

   We are currently investigating an issue impacting virtual machine accessibility. Some users may be unable to stop and start Virtual Machines.

2. monitoring Jun 11, 2025, 03:20 AM UTC

   We have identified the issue and a fix has been implemented. Currently, there is no further impact expected to virtual machine accessibility. We are currently monitoring the environment.

3. resolved Jun 11, 2025, 03:36 AM UTC

   We are marking this incident as resolved. Please reach out to [email protected] if you experience any issues.

Read the full incident report →

Started May 30, 2025, 01:05 PM UTC· resolved May 30, 2025, 04:52 PM UTC

1. investigating May 30, 2025, 01:05 PM UTC

   We are currently investigating a potential service interruption that may impact some customers in our us-east1-a region. If you are experiencing issues in this region, please reach out to [email protected]

2. investigating May 30, 2025, 01:18 PM UTC

   We are continuing to investigate this issue.

3. investigating May 30, 2025, 01:32 PM UTC

   We are currently investigating an issue impacting virtual machine accessibility in us-east1-a region. Some users may be unable to access their VMs at this time. We appreciate your patience and will provide updates as more information becomes available.

4. investigating May 30, 2025, 01:33 PM UTC

   We are continuing to investigate this issue.

5. investigating May 30, 2025, 01:34 PM UTC

   We are continuing to investigate this issue.

6. investigating May 30, 2025, 01:44 PM UTC

   We are continuing to investigate this issue.

7. investigating May 30, 2025, 03:24 PM UTC

   We're making significant progress in resolving the virtual machine accessibility issues in the us-east1-a region. We apologize for the continued disruption and thank you for your patience as we work to restore full service. We will provide another update as soon as more information is available or when services are fully restored. You can continue to contact our support team at [email protected], if there are any other concerns to address

8. monitoring May 30, 2025, 03:55 PM UTC

   We have identified and mitigated the underlying cause. Although no further impact is expected, we are continuing to monitor. If you are still experiencing issues, please reach out to us through [email protected] and we will investigate further.

9. monitoring May 30, 2025, 03:55 PM UTC

   We are continuing to monitor for any further issues.

10. monitoring May 30, 2025, 04:45 PM UTC

    A fix has been applied and the outage has been mitigated. Services have returned to normal operations in us-east1-a region. We are continuing to monitor the situation and validate preventive measures. Please contact [email protected] if you are still experiencing any issues.

11. resolved May 30, 2025, 04:52 PM UTC

    This incident has been resolved.

Read the full incident report →