Crusoe Outage History

Started Oct 15, 2026, 01:12 PM UTC· resolved Oct 18, 2026, 08:22 PM UTC

1. investigating Oct 15, 2026, 01:12 PM UTC

   We’re investigating an issue at one of our data center facilities that is affecting a subset of compute instances in us-east2-a. Some instances are currently unavailable. As a safety precaution, we’re powering down some additional hardware in the affected area. Instances on that hardware will become unavailable during this time. Our teams are actively working on this, and we’ll post another update within 30 minutes.

2. investigating Oct 15, 2026, 01:50 PM UTC

   As a safety precaution, we’ve shut off power to the affected hardware in us-east2-a. Instances running on that hardware are now powered off. Power won’t be restored until the area has been confirmed safe. Our teams are working on recovery, and we’ll post another update within 60 minutes.

3. identified Oct 15, 2026, 04:03 PM UTC

   We’ve started restoring power to part of the affected hardware in us-east2-a, and some instances will begin to come back online. The rest of the affected hardware will stay powered off while the facility issue is being remediated. Our teams are actively working on recovery, and we’ll post updates as events warrant.

4. identified Oct 15, 2026, 07:20 PM UTC

   Recovery work for the facility issue in us-east2-a is ongoing. The remaining affected hardware will stay powered off until required facility work and safety inspections are complete. We don’t have an estimated restoration time yet, and we’ll share one as soon as those inspections are done. In the meantime, we’re restoring affected instances on other available capacity in us-east2-a where possible, and a number of instances are already back online. If your instances are affected, our customer success team is in contact with you directly. We’ll post another update by 23:00 UTC

5. identified Oct 16, 2026, 12:59 AM UTC

   Work to address the facility issue in us-east2-a continues to progress. The hardware that's still affected will stay offline until facility repairs and safety checks are finished, and we don't yet have a time for full restoration. Our teams are still moving impacted instances onto other available capacity in us-east2-a where we can. We will post updates on further developments within the next 24 hours.

6. identified Oct 16, 2026, 07:22 AM UTC

   We have restored part of the affected capacity in us-east2-a and continue to move impacted instances onto other available capacity where we can. The remaining affected hardware will stay offline until facility repairs and safety checks are finished, and we don’t yet have a time for full restoration. We will post an update in 24 hours, or as events warrant.

7. identified Oct 16, 2026, 02:48 PM UTC

   Work to address the facility issue in us-east2-a continues. The affected hardware will stay offline until facility repairs and safety checks are complete, and it will be brought back online as that work allows. We don’t yet have a time for full restoration. Our teams continue to move impacted instances onto other available capacity in us-east2-a where we can. We will post an update in 24 hours, or as events warrant.

8. identified Oct 17, 2026, 07:16 AM UTC

   We have restored additional capacity in us-east2-a and are validating each host before returning it to service. Some affected customers are now fully restored, and we continue to return capacity to the remaining impacted customers. The remaining affected hardware will stay offline until facility repairs and safety checks are finished, and we don’t yet have a time for full restoration. We will post an update in 24 hours, or as events warrant.

9. identified Oct 18, 2026, 04:51 PM UTC

   Most affected instances in our Ohio region are back in service. A small number are still unavailable while we finish hardware checks and the last repairs. We’ll keep you posted on further developments

10. resolved Oct 18, 2026, 08:22 PM UTC

    This incident has been resolved.

Read the full incident report →

Started Sep 29, 2026, 08:38 PM UTC· resolved Sep 30, 2026, 12:32 AM UTC

1. investigating Sep 29, 2026, 08:38 PM UTC

   We are currently investigating storage accessibility issues when using Shared Storage and Object Storage in eu-norway1. We are working with our storage vendor to resolve the issue as soon as possible and will keep you posted on further developments.

2. identified Sep 29, 2026, 09:18 PM UTC

   We are seeing early signs of recovery for Shared Storage, Object Storage and Crusoe Container Registry in eu-norway1. We are still working closely with our storage vendor to fully restore service and will share another update as progress continues.

3. monitoring Sep 29, 2026, 11:43 PM UTC

   Shared Storage, Object Storage and Crusoe Container Registry in eu-norway1 are available again. We have put a fix in place with our storage vendor and are monitoring the results. We will share another update once we confirm the service is stable.

4. resolved Sep 30, 2026, 12:32 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 09, 2026, 09:56 AM UTC· resolved Sep 09, 2026, 11:19 AM UTC

1. identified Sep 09, 2026, 09:56 AM UTC

   We're currently investigating an issue affecting a subset of customer VMs in our us-east2 region . A subset of VMs are showing as UNSPECIFIED in our control plane . We have identified the issue and are actively working on fixing it to bring affected VMs back to a healthy state. We will provide more information as soon as it becomes available.

2. monitoring Sep 09, 2026, 10:51 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 09, 2026, 11:19 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 03, 2026, 12:30 PM UTC· resolved Sep 03, 2026, 12:30 PM UTC

1. resolved Sep 03, 2026, 02:24 PM UTC

   We identified general network errors affecting service connectivity in our us-east2 region. The issue has been resolved and services are operating normally.

Read the full incident report →

Started Sep 02, 2026, 08:22 AM UTC· resolved Sep 02, 2026, 03:57 PM UTC

1. investigating Sep 02, 2026, 08:22 AM UTC

   We are investigating "insufficient capacity" errors when creating VMs or CMK clusters in us-east2 region. Existing VMs continue to run normally. We'll share an update as soon as we have more information available.

2. investigating Sep 02, 2026, 09:44 AM UTC

   We continue investigating "insufficient capacity" errors when creating VMs or CMK clusters in us-east2 region, and have narrowed down a likely cause. We'll share an update as soon as we have more information available.

3. investigating Sep 02, 2026, 11:07 AM UTC

   We continue investigating "insufficient capacity" errors when creating VMs or CMK clusters in us-east2 region. We'll share an update as soon as we have more information available.

4. investigating Sep 02, 2026, 12:56 PM UTC

   We continue investigating "insufficient capacity" errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region. We'll share an update as soon as we have more information available.

5. monitoring Sep 02, 2026, 01:40 PM UTC

   We've identified the cause of the errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region and applied a fix. We'll continue monitoring.

6. resolved Sep 02, 2026, 03:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 09, 2026, 02:45 PM UTC· resolved Jul 09, 2026, 04:58 PM UTC

1. investigating Jul 09, 2026, 02:45 PM UTC

   We are currently investigating reports of degraded network connectivity in our us-east-1-a region. Some customers may be experiencing connection timeouts, SSL handshake failures, or degraded network performance for workloads hosted in this region. Our engineers are actively working to mitigate the issue as soon as possible. We will provide further updates as more information becomes available.

2. investigating Jul 09, 2026, 03:10 PM UTC

   We have implemented a fix for the network connectivity issues affecting us-east-1-a. We are seeing improvements across impacted services and are actively monitoring the environment to confirm full recovery. We will provide a further update once we have verified complete resolution. We apologize for any disruption this has caused.

3. monitoring Jul 09, 2026, 03:12 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 09, 2026, 04:58 PM UTC

   This incident has been resolved. Network connectivity in our us-east-1-a region has been fully restored. We sincerely apologize for any disruption this may have caused and thank you for your patience.

Read the full incident report →

Started Jul 08, 2026, 05:23 AM UTC· resolved Jul 09, 2026, 03:23 PM UTC

1. investigating Jul 08, 2026, 05:23 AM UTC

   We are investigating an issue affecting the Crusoe Container Registry (CCR) in the US South Central (us-southcentral1-a) region. Beginning at approximately 04:07 UTC on July 8, 2026, customers may be unable to push or pull container images, and may see TLS handshake timeouts or authentication errors. Our engineering team is actively investigating. We will provide more information as we investigate the issue.

2. investigating Jul 08, 2026, 11:03 AM UTC

   We continue to investigate the issue and have narrowed down the root cause. Our team is actively working towards a resolution and will provide a further update as soon as possible.

3. monitoring Jul 08, 2026, 09:57 PM UTC

   We have applied a fix which looks to have mitigated the issue. Please let us know if you are still seeing an issue. We are continuing to monitor.

4. resolved Jul 09, 2026, 03:23 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 25, 2026, 09:30 AM UTC· resolved Jun 25, 2026, 09:30 AM UTC

1. resolved Jun 25, 2026, 11:33 PM UTC

   A subset of resources in us-east1-a experienced intermittent external connectivity disruption during a planned network maintenance window. Affected workloads may have observed elevated latency, connection timeouts, or failures reaching external destinations. Virtual machine availability was not affected. The change was rolled back to restore connectivity, which was confirmed stable following the revert.

Read the full incident report →

Started Jun 16, 2026, 09:15 PM UTC· resolved Jun 17, 2026, 01:17 AM UTC

1. investigating Jun 16, 2026, 09:15 PM UTC

   Beginning at approximately 13:55 PDT, load balancers in the ICAT region became unreachable. Customers relying on load balancer endpoints in this region may be experiencing connectivity failures. Our team is actively investigating and working to restore service. We will provide updates as more information becomes available.

2. investigating Jun 17, 2026, 01:17 AM UTC

   The issue was identified and resolved as of approximately 15:45 PDT. Load balancer connectivity in the ICAT region has been restored.

3. resolved Jun 17, 2026, 01:17 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 11, 2026, 04:50 PM UTC· resolved Jun 11, 2026, 07:47 PM UTC

1. investigating Jun 11, 2026, 04:50 PM UTC

   We have identified an issue affecting custom image creation in us-east1-a, us-southcentral1-a and eu-iceland1-a regions. We have identified the underlying cause and are actively working to restore service. We will provide more information as soon as possible.

2. identified Jun 11, 2026, 04:51 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Jun 11, 2026, 07:47 PM UTC

   This issue is now resolved.

Read the full incident report →

Started Jun 08, 2026, 10:12 PM UTC· resolved Jun 09, 2026, 02:56 PM UTC

1. investigating Jun 08, 2026, 10:12 PM UTC

   We are seeing a network outage in this region, which is currently under investigation. We will keep you posted on further developments as we make progress on this matter

2. monitoring Jun 08, 2026, 11:41 PM UTC

   A fix has been implemented and we are monitoring. Please let us know if you are still seeing any network related performance issues. We will keep you updated with any further developments.

3. resolved Jun 09, 2026, 02:56 PM UTC

   This issue has been resolved.

Read the full incident report →

Started Jun 04, 2026, 02:10 AM UTC· resolved Jun 04, 2026, 04:27 AM UTC

1. investigating Jun 04, 2026, 02:10 AM UTC

   We are seeing intermittent network degradation in this region, which is currently under investigation. We will keep you posted on further developments as we make progress on this matter

2. monitoring Jun 04, 2026, 03:31 AM UTC

   We are now monitoring the incident and if you see any issues, please reach out to support.

3. resolved Jun 04, 2026, 04:27 AM UTC

   This issue is now resolved.

Read the full incident report →

Started May 25, 2026, 12:32 PM UTC· resolved May 26, 2026, 12:26 AM UTC

1. investigating May 25, 2026, 12:32 PM UTC

   We are investigating an issue on the eu-iceland1-storage cluster. Customers may experience elevated write latency. We will provide an update as soon as we have more information available.

2. investigating May 25, 2026, 01:08 PM UTC

   We are continuing to investigate this issue.

3. investigating May 25, 2026, 02:28 PM UTC

   We are continuing to investigate this issue.

4. identified May 25, 2026, 03:15 PM UTC

   We are seeing an improvement in latency and our teams remain fully focused on driving further recovery. We will provide an update as soon as we have more information available.

5. resolved May 26, 2026, 12:26 AM UTC

   This issue has been resolved.

Read the full incident report →

Started May 20, 2026, 01:37 AM UTC· resolved May 20, 2026, 09:09 PM UTC

1. investigating May 20, 2026, 01:37 AM UTC

   We are investigating a networking issue in the us-east1-a region affecting connectivity to a subset of compute hosts. Affected virtual machines may appear in an unspecified state or be unreachable over the network. Managed Kubernetes clusters in the region may also report nodes as Not Ready. Engineering is engaged and actively working to identify the cause and restore service. We will provide updates as we have more information.

2. investigating May 20, 2026, 03:51 AM UTC

   We are continuing to investigate this issue.

3. identified May 20, 2026, 06:19 AM UTC

   We are continuing to work on the issue affecting us-east1-a. Engineering has identified a likely cause and mitigation is in progress. A subset of compute hosts remains unavailable, and Managed Kubernetes and Container Registry operations in the region continue to be impacted. We will continue to provide updates as remediation progresses.

4. identified May 20, 2026, 08:25 AM UTC

   We are seeing partial recovery in us-east1-a following mitigation steps applied by engineering. A subset of affected virtual machines have returned to service. Engineering is continuing to validate the mitigation and work toward full restoration. A subset of compute hosts remains impacted at this time. We will provide further updates as progress continues.

5. identified May 20, 2026, 09:28 AM UTC

   We continue to see progress in us-east1-a as engineering works through mitigation. Additional virtual machines have returned to service, although a subset of compute hosts remains impacted, and the issue is not yet fully mitigated. Engineering remains actively engaged, and we will provide further updates as recovery continues.

6. identified May 20, 2026, 10:32 AM UTC

   Remediation in us-east1-a is ongoing and continuing to progress. Additional virtual machines have returned to service; however, a subset of compute hosts remains impacted and the region is not yet fully restored. Engineering remains engaged and further updates will follow.

7. identified May 20, 2026, 11:28 AM UTC

   Engineering continues to work on the issue in us-east1-a. A subset of compute hosts in the region remains impacted while remediation steps are being applied. We will provide a further update once additional progress can be reported

8. identified May 20, 2026, 12:37 PM UTC

   Work to restore service is ongoing as our teams continue to drive remediation in us-east1-a. Engineering remains engaged across affected compute hosts and we will provide a further update shortly.

9. identified May 20, 2026, 01:27 PM UTC

   Engineering continues to make progress on the issue in us-east1-a, though recovery is proceeding more slowly than initially anticipated. A subset of compute hosts in the region remains impacted while remediation work continues. We will provide a further update as recovery advances.

10. identified May 20, 2026, 02:27 PM UTC

    Work to restore service in us-east1-a continues, with engineering making steady progress across affected compute hosts. A subset of hosts remain impacted as remediation work advances. We will provide a further update shortly.

11. identified May 20, 2026, 03:35 PM UTC

    Work to restore service in us-east1-a continues. Engineering has made significant progress, with the large majority of affected compute hosts now recovered. A small number of hosts remain under active remediation. We will provide a further update shortly.

12. identified May 20, 2026, 04:30 PM UTC

    Engineering has restored the majority of affected services in us-east1-a and is in the final stages of remediation. A subset of customers remain impacted and are being prioritised for recovery. We will provide a further update shortly.

13. identified May 20, 2026, 05:55 PM UTC

    Engineering continues to make progress on remediation since the last status post. Work remains ongoing for the customers still in progress, We will provide a further update as recovery advances.

14. identified May 20, 2026, 06:55 PM UTC

    Remediation is ongoing and the number of impacted customers continues to narrow. Engineering remains actively engaged, and we will provide a further update as the work advances.

15. resolved May 20, 2026, 09:09 PM UTC

    This incident has been resolved. All affected compute hosts and customer virtual machines in us-east1-a have been restored.

Read the full incident report →

Started May 18, 2026, 08:24 AM UTC· resolved May 19, 2026, 01:19 AM UTC

1. identified May 18, 2026, 08:24 AM UTC

   We are investigating an issue affecting the ability to create and start Virtual Machines in the us-southcentral1-a and eu-iceland1-a regions. Existing running workloads and resources remain fully operational and are not impacted. Our Engineering teams are actively working on recovery. We will provide an update when more information is available.

2. identified May 18, 2026, 11:27 AM UTC

   We have identified the underlying issue affecting the ability to create and start Virtual Machines in the us-southcentral1-a and eu-iceland1-a regions. Our Engineering teams are actively working on recovery. Some customers may still be experiencing impact while this work continues. Existing running workloads and resources remain fully operational and are not impacted. We will provide a further update as soon as more information becomes available.

3. monitoring May 18, 2026, 02:34 PM UTC

   The issue affecting the ability to create and start Virtual Machines in the us-southcentral1-a and eu-iceland1-a regions has been mitigated. VM creation and start operations are functioning normally. No customer impact has been observed as of 14:00 UTC. Our Engineering teams continue to monitor both regions closely and are working on a permanent fix to prevent recurrence. We apologize for any inconvenience this may have caused.

4. resolved May 19, 2026, 01:19 AM UTC

   VM creation and start operations in the us-southcentral1-a and eu-iceland1-a regions have been functioning normally for several hours. Our Engineering teams have addressed the underlying issue and a permanent fix is being prepared for deployment to prevent recurrence. Existing running workloads and resources remained fully operational throughout the incident. We apologize for any inconvenience this may have caused.

Read the full incident report →

Started May 13, 2026, 09:34 PM UTC· resolved May 14, 2026, 12:09 AM UTC

1. investigating May 13, 2026, 09:34 PM UTC

   Customers using CCR (Crusoe Container Registry) will be unable to push or pull images. We are working on resolving this matter as soon as possible. If you experience any other issues, please reach out to [email protected]

2. resolved May 14, 2026, 12:09 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 10, 2026, 02:41 PM UTC· resolved May 10, 2026, 06:19 PM UTC

1. investigating May 10, 2026, 02:41 PM UTC

   We are investigating an issue affecting access to Crusoe Managed Inference, including Intelligence Foundry and Bring Your Own Model (BYOM). Customers may be unable to reach inference endpoints or send requests to the service. Engineering is actively working on recovery. We will provide an update when more information is available.

2. resolved May 10, 2026, 06:19 PM UTC

   Service has been fully restored. We have identified the root cause and are taking steps to prevent recurrence.

Read the full incident report →

Started May 08, 2026, 08:04 PM UTC· resolved May 09, 2026, 03:09 AM UTC

1. investigating May 08, 2026, 08:04 PM UTC

   We're investigating failures when creating new projects. Existing workloads and resources remain fully operational. We will provide an update when more information is available.

2. investigating May 09, 2026, 03:09 AM UTC

   The issue has been mitigated and project creation requests are completing successfully.

3. resolved May 09, 2026, 03:09 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 06, 2026, 10:19 PM UTC· resolved May 06, 2026, 11:41 PM UTC

1. investigating May 06, 2026, 10:19 PM UTC

   We are currently experiencing an issue with storage used disk capacity metrics in our Iceland (eu-iceland1) region. All data access, read/write operations, and service functionality continue to operate normally with no performance impact. Our team is actively working to restore capacity metrics. We will provide updates as restoration progresses.

2. resolved May 06, 2026, 11:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 04, 2026, 10:26 PM UTC· resolved May 04, 2026, 11:29 PM UTC

1. monitoring May 04, 2026, 10:26 PM UTC

   We are investigating an issue causing 503 errors when accessing the Crusoe Compute Console and API. Users may currently be unable to list instances or manage infrastructure resources. Our engineering team is actively working to identify the root cause. Further updates will be provided as soon as they are available.

2. monitoring May 04, 2026, 10:26 PM UTC

   The issues affecting the Crusoe Compute Console, API, and CLI have been fully resolved. Impact Window: The service interruption lasted from 2:42 PM PDT to 3:16 PM PDT. During this window, users may have experienced 503 errors and were unable to view or manage resources via the web console, CLI, or API. Our engineering team identified the issue and implemented a fix to restore all management services. We have verified that all systems are now performing as expected. We are continuing to monitor the platform and will implement additional safeguards and alerting to improve our response and stability moving forward. Note: Active customer workloads and existing infrastructure remained operational and were not impacted by this management-layer interruption.

3. resolved May 04, 2026, 11:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 17, 2026, 01:30 AM UTC· resolved Apr 17, 2026, 02:52 AM UTC

1. investigating Apr 17, 2026, 01:30 AM UTC

   We are currently investigating an issue specifically affecting Persistent Volume (PVC) creation for Crusoe Managed Kubernetes (CMK) customers. While this interruption prevents the provisioning of new block storage volumes, it is strictly limited to block storage and does not impact Shared Disks or other services. Existing volumes and active workloads remain fully operational and unaffected. Our engineering team has identified the root cause and is restoring the service. We expect to provide a final update once provisioning functionality is fully restored.

2. resolved Apr 17, 2026, 02:52 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 01, 2026, 01:29 PM UTC· resolved Apr 01, 2026, 02:05 PM UTC

1. investigating Apr 01, 2026, 01:29 PM UTC

   We are investigating an issue in our eu-iceland1 region that may be impacting the ability to manage resources. Our engineering team is actively working to identify the scope and cause. We will provide another update as more information becomes available.

2. monitoring Apr 01, 2026, 01:51 PM UTC

   A fix has been implemented and we are monitoring the results. If you are experiencing issues, please contact Crusoe Support.

3. resolved Apr 01, 2026, 02:05 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 30, 2026, 09:12 PM UTC· resolved Mar 30, 2026, 11:14 PM UTC

1. investigating Mar 30, 2026, 09:12 PM UTC

   We are aware of an issue with an upstream dependency that may impact the ability to add new nodes to existing node pools. Existing clusters and running workloads are not affected. We are actively monitoring the situation and will provide updates as it progresses. If you are experiencing issues, please contact Crusoe Support.

2. resolved Mar 30, 2026, 11:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 18, 2026, 08:21 PM UTC· resolved Mar 18, 2026, 11:30 PM UTC

1. investigating Mar 18, 2026, 08:21 PM UTC

   We are investigating an issue affecting Kubernetes CMK control plane operations. Customers may experience elevated latency or timeouts when retrieving cluster credentials via the CLI or console, and may be unable to create new clusters. Existing cluster workloads are not affected. We will provide a further update as our investigation continues.

2. resolved Mar 18, 2026, 11:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 04, 2026, 07:36 PM UTC· resolved Mar 04, 2026, 09:25 PM UTC

1. investigating Mar 04, 2026, 07:36 PM UTC

   We are investigating an issue in our US-Southcentral region that may be impacting the ability to manage cluster resources via the API. Our engineering team is actively working to identify the scope and cause. We will provide another update as more information becomes available.

2. investigating Mar 04, 2026, 09:25 PM UTC

   We are pleased to report that this incident is now resolved. We have confirmed that control plane functionality for all impacted clusters in the us-southcentral region has been fully restored. The issue was caused by network congestion on a small number of underlying physical hosts. This has been remediated by isolating the source of the traffic. We will be conducting a full post-mortem to prevent a recurrence and apologize for any disruption this incident caused.

3. resolved Mar 04, 2026, 09:25 PM UTC

   This incident has been resolved.

Read the full incident report →