Adeptcore Outage History

Started Jul 10, 2026, 10:34 AM UTC· resolved Jul 23, 2026, 07:44 PM UTC

1. investigating Jul 10, 2026, 10:34 AM UTC

   We are receiving reports of a widespread outage across our PHX Datacenter and its VM's. We are currently investigating this issue and interfacing with datacenter engineers to find the issue and get this resolved as quickly as possible. We will update this status as more information becomes available.

2. investigating Jul 10, 2026, 11:07 AM UTC

   We are continuing to investigate the issue affecting our PHX datacenter. Preliminary indicators suggest the incident may be related to either networking and/or power infrastructure. Our engineering team is actively working alongside datacenter personnel to isolate the root cause and restore normal operations as quickly as possible.

3. investigating Jul 10, 2026, 11:43 AM UTC

   A datacenter engineer is currently on-site at the affected rack, and we are actively investigating the issue together. We have confirmed access to our out-of-band management systems, which indicates external connectivity to our Cisco network infrastructure. However, traffic is not currently passing beyond that point into the environment.

4. investigating Jul 10, 2026, 12:27 PM UTC

   Our investigation has confirmed that both power and network connectivity within the datacenter are operating normally. We are now focusing on what appears to be a potential storage-related issue. This is proving to be a complex incident, and we are continuing to work closely with datacenter engineers to isolate the root cause and restore service as quickly as possible. We understand the impact this is having and appreciate your patience. We will provide additional updates as soon as more information becomes available.

5. investigating Jul 10, 2026, 01:19 PM UTC

   We continue to actively investigate this issue in collaboration with the datacenter engineering team. At this time, we do not yet have an estimated time for service restoration. Our teams remain engaged in identifying the root cause and implementing the appropriate remediation as quickly as possible. We understand the impact this outage is having and appreciate your continued patience. We will provide another update as soon as additional information becomes available.

6. investigating Jul 10, 2026, 01:47 PM UTC

   We continue to work alongside datacenter engineers to investigate the root cause of this incident. Our teams remain actively engaged in troubleshooting and evaluating all potential causes. At this time, we do not have an estimated time for service restoration. We understand the impact this outage is having and appreciate your continued patience as we work to resolve the issue. We will provide another update as soon as additional information becomes available.

7. investigating Jul 10, 2026, 02:19 PM UTC

   We are continuing to work with the datacenter engineering team to resolve this incident. We understand the frustration caused by the extended disruption and sincerely appreciate your patience as our teams work through this issue. Please know that we are doing everything we can to restore service as quickly and safely as possible. We remain actively engaged in the investigation and will continue to provide updates as new information becomes available.

8. investigating Jul 10, 2026, 03:36 PM UTC

   We have identified an issue affecting our storage arrays and are actively working to restore them to a previous snapshot as part of our investigation and recovery efforts. This process will help us further isolate the underlying issue and validate system integrity before restoring services. We understand the significant impact this outage is having and sincerely appreciate your patience. At this time, we are unable to provide an estimated time for completion of the restoration and verification process. We will continue to share updates as additional information becomes available.

9. investigating Jul 10, 2026, 04:25 PM UTC

   We are currently working through the process of mounting and unmounting storage volumes as part of our recovery efforts. This is a time-intensive process, and we are carefully validating each step to ensure data integrity and a safe restoration of services. We appreciate your continued patience while this work is underway. We will provide additional updates as significant progress is made or within the next two hours, whichever comes first.

10. investigating Jul 10, 2026, 06:16 PM UTC

    We continue to make progress on the storage recovery process. Early indicators remain encouraging, and the restoration and validation activities are progressing as expected. To expedite resolution, we have dedicated one team to restoring storage services while a second team continues investigating the underlying cause of the incident. This parallel approach allows us to restore service as quickly as possible while ensuring the root cause is fully understood and addressed. We remain committed to restoring services safely and as quickly as possible. We sincerely appreciate your continued patience and will provide additional updates as significant progress is made or within the next two hours, whichever comes first.

11. investigating Jul 10, 2026, 08:40 PM UTC

    During our ongoing investigation, we identified indicators consistent with a potential compromise involving one of our dedicated storage servers. Upon identifying these indicators, we immediately isolated the affected management system and associated storage infrastructure to contain the issue and minimize any further potential impact. The machine believed to be the source of the activity is now fully isolated and is being retained for further investigation. We have also reviewed other systems and, at this time, have not identified any additional indicators of compromise. However, the investigation remains ongoing, and we have not yet fully determined the scope or nature of the incident. We are working closely with our datacenter partner to restore storage systems from previously validated snapshots. As part of this process, we are currently restoring a single datastore to validate the recovery procedure, confirm the integrity of the restored data, and help us better assess the impact before proceeding with additional storage systems. Based on our investigation, we believe the networking symptoms observed earlier were related to activity originating from the affected management virtual machine. Our current focus is on safely restoring services while continuing to investigate the root cause and confirm the integrity of the affected environment. We understand the impact this outage is having and appreciate your continued patience. We will provide additional updates as recovery progresses and more information becomes available.

12. investigating Jul 10, 2026, 09:34 PM UTC

    Hello Partners, I want to personally provide an update regarding the major outage affecting one of our data center environments. I also want to be as clear and transparent as possible about the situation and the recovery work currently underway. Early this morning, we identified multiple servers that had lost network connectivity. We immediately engaged both the data center’s network engineering team and our internal network engineering team to begin investigating. Throughout the day, we have systematically isolated and tested the components of our network, storage infrastructure, and virtual environments. During the investigation, we identified activity that required us to treat the situation as a potential security incident. Out of an abundance of caution, I made the decision to take the storage network offline and begin disaster recovery operations. Our teams are currently moving more than 100 TB of data across multiple storage arrays and restoring systems from point-in-time snapshots. This is a significant recovery operation that requires careful validation at each stage to ensure the integrity and safety of the restored data. We are currently testing backups from approximately 10:00 a.m. Central Time today, July 10, 2026, as well as backups from approximately 10:00 a.m. Central Time yesterday, July 9, 2026. At this time, we have not confirmed whether a breach occurred. In parallel with the recovery effort, we are conducting extensive log reviews and analysis across our Phoenix clusters, networks, storage systems, and supporting infrastructure to determine the full scope and nature of the incident. No one wants to begin their Friday with a service disruption of this magnitude. We understand the significant impact this outage has on our partners and their customers. We have a dedicated team working continuously on recovery operations, and I am personally involved in coordinating and overseeing this process. Because of the volume of data involved, recovery and validation are taking time. Based on our current progress, we expect to begin bringing virtual machines and services back online overnight and into Saturday morning, July 11, 2026. This remains an estimate and may change based on the results of our data validation and security review. Due to the widespread nature of the recovery effort, we cannot safely restore individual virtual machines one at a time or prioritize a single system independently. The storage, network, and virtualization layers are interconnected, and we must ensure the entire environment is restored in a controlled manner that protects data integrity and prevents further disruption. We expect to provide additional updates every few hours as storage volumes are migrated, restored, tested, and validated. AdeptCloud appreciates the patience and understanding of every partner affected by this incident. We sincerely apologize for the disruption and will continue to communicate openly as more information becomes available. Denis Zhirovetskiy President AdeptCloud

13. investigating Jul 11, 2026, 01:37 AM UTC

    Our recovery efforts are continuing and will remain active throughout the night as data restoration progresses. To support validation, we performed immediate volume recoveries into isolated environments. These recovered volumes have now been running for several hours with a limited number of isolated virtual machines powered on for testing. We have also cloned selected environments and data from test systems to further validate stability and integrity. The next stages of recovery will include network configuration, auditing, and continued validation of restored volumes. Progress remains encouraging, with approximately 50 TB of data verified so far. We still have approximately 50–60 TB of data to move and are actively monitoring the restoration and validation processes as they continue. Our teams remain fully engaged, and we will continue to provide updates as additional progress is made.

14. investigating Jul 11, 2026, 10:14 AM UTC

    Hello Partners, I want to provide a continued update regarding the outage affecting one of our data center environments and the progress of our recovery efforts. Our teams have remained fully engaged throughout the recovery process. We have now restored approximately 120 TB of data from July 9 and approximately 30 TB of data from July 10. All restored volumes and data are currently showing as healthy through our validation processes, and we remain actively engaged in continued data validation to ensure integrity and stability across the environment. In parallel with data restoration, we have begun network reconstruction efforts. Our networking team is actively rebuilding and verifying WAN configurations and related network settings to ensure a stable and secure environment as services are brought back online. With confidence in the stability and security of the network and restored data, we have begun bringing client environments back online. Overnight, we initiated the restoration of customer virtual machines for testing and confirmation. To date, we have successfully brought up and verified approximately 150 customer virtual machines, confirming both operational functionality and data integrity. Our teams have been working continuously through the night and will continue throughout the day to expand restoration efforts. Based on current progress, we expect customers to begin regaining access to their environments later today, continuing through the evening. As part of our ongoing investigation, we have identified activity within portions of our underlying infrastructure that required us to take precautionary measures, including isolating systems and initiating recovery procedures. At this time, there is no indication that client data was accessed or removed from our environment, and our analysis confirms that no data exfiltration occurred. Our review of system logs and traffic patterns continues as we work to fully understand the scope and nature of the incident. We understand the impact this outage has had on our partners and their customers, and we sincerely appreciate your continued patience as we work through the final stages of recovery. We will continue to provide updates as additional progress is made. Denis Zhirovetskiy President AdeptCloud

15. investigating Jul 12, 2026, 02:38 AM UTC

    We are currently bringing customer environments back online in a controlled and measured manner. This process is proceeding as planned, but it will take time to complete as we carefully manage the workload to avoid overloading the storage arrays. Our teams are actively monitoring the environment throughout the recovery process. We will continue to provide updates as additional customer environments are restored and progress is made.

16. investigating Jul 12, 2026, 03:56 PM UTC

    Customer environments operating behind our physical networking infrastructure have been brought back online. We are continuing to repair and validate the virtual networking configurations. We anticipate completing the virtual networking restoration today, after which we can begin bringing the virtual machines connected to those networks back online in a controlled manner. We appreciate your continued patience and will provide additional updates as restoration progresses.

17. investigating Jul 13, 2026, 01:39 AM UTC

    We are continuing to restore client systems, with additional customers being brought back online on an ongoing basis. During the recovery process, we identified an issue affecting our virtual networking infrastructure that requires us to manually rebuild customer WAN configurations. Our teams have been working continuously throughout the weekend and will remain engaged through the night to restore connectivity for as many customers as possible before Monday morning. We sincerely apologize for the continued disruption and delays. We appreciate your patience as we work to bring all remaining customer environments back online as quickly and safely as possible. We will continue to provide updates as restoration progresses.

18. investigating Jul 13, 2026, 04:58 AM UTC

    We have restored connectivity for the majority of our clients and virtual machines. Our teams are continuing to trace and resolve the remaining network issues so that the rest of the affected environments can be brought back online. We remain actively engaged in the restoration process and will continue to provide updates as additional progress is made.

19. identified Jul 13, 2026, 01:21 PM UTC

    We have restored connectivity for the majority of our clients and virtual machines, and the environment continues to stabilize. Our teams remain actively engaged in resolving the remaining issues. At this time, we are still seeing isolated pockets of network configuration issues affecting a subset of customer environments, along with a number of individual servers that require additional attention. As we work through the recovery process, we continue to encounter small, one-off issues affecting some virtual machines, and resolving these is taking time. Our teams are addressing them individually as they arise to bring each affected environment back online. Once the current issues are resolved, we will be conducting a full review of all virtual machines both those previously running and those currently running from snapshots to identify and complete any remaining recovery actions still available to us. We have also received requests from some clients for copies of their individual backups. Please be aware that we are not able to fulfill these requests at this time, as we are still actively working through the restoration process and completing our assessment of the environment. We appreciate your understanding and will provide guidance on this once recovery is further along. We do not have a precise completion time for all remaining items, as this remains an active recovery effort, but our teams remain engaged continuously until every environment is fully restored. We sincerely appreciate your continued patience as we work through the final stages of recovery. We will continue to provide updates as additional progress is made.

20. monitoring Jul 14, 2026, 09:14 PM UTC

    What Happened An unauthorized third party gained access to an externally accessible system within our data center environment and deployed ransomware that affected a portion of our hosted virtual infrastructure. Once the activity was identified, we contained the affected systems and immediately began our incident response and recovery process. Our investigation into the exact method of initial access is still ongoing. While that investigation and the related remediation work continue, we are intentionally not publishing the specific technical entry point. This is necessary to protect the environment during recovery and to avoid stating a cause before it has been fully validated. We are committed to providing a more detailed technical account once the investigation is complete and all required remediation has been implemented. What This Means for Your Data We understand that customers have two primary concerns: whether their data was accessed or stolen, and whether their systems and data can be recovered. Data Exposure As part of our investigation, we reviewed the network traffic leaving the affected environment during the incident. Based on the volume of outbound traffic, the available connection throughput, and our review of the relevant network and security logs, we found no evidence that customer data was copied or exfiltrated from the environment. Our investigation remains ongoing, and we will communicate directly with affected customers if any new information changes this assessment. System and Data Recovery The majority of affected systems have been, or are currently being, restored from clean backups and storage snapshots created before the incident. These systems are being returned to a known good state from before the ransomware activity occurred. A small number of systems require additional recovery work. We are handling those systems individually and communicating directly with the affected customers and partners regarding their recovery status. Actions We Have Taken We responded immediately once the unauthorized activity was identified. Contained the Affected Systems Our first action was to take the impacted systems offline. This halted the active ransomware process and prevented further spread within the environment. Reviewed System and Security Logs We conducted an extensive review of system, network, storage, virtualization, and security logs to determine what occurred, how the activity progressed, and which systems were affected. Removed Unauthorized Access We removed the unauthorized party’s access to the environment and eliminated the tools, accounts, processes, and changes associated with the incident. Initiated a Full Environment Restoration Rather than attempting to clean and return the affected systems directly to service, we began rebuilding and restoring the environment from clean backups and storage snapshots that predated the incident. This approach allows us to return systems to a known good state and reduces the risk of leaving behind unauthorized tools, changes, or persistence mechanisms. Reset Credentials We reset credentials across the affected environment and reviewed privileged access to infrastructure, management systems, storage, and virtualization platforms. Reviewed Outbound Network Activity We analyzed network activity associated with the incident to determine whether customer data left the environment. Based on the evidence reviewed to date, we have found no indication that customer data was exfiltrated. Current Status We continue to work directly with a small number of partners whose systems require additional restoration or recovery work. Our team is supporting those partners throughout the process and providing individual updates as new information becomes available. Once the investigation, recovery, and remediation work are complete, we intend to publish a more detailed summary of the incident and the changes we have made to strengthen the security and resilience of our environment.

21. monitoring Jul 15, 2026, 07:54 PM UTC

    Service Recovery Update Our team continues to work through the remaining recovery efforts related to the ransomware incident affecting a portion of our hosted environment. The majority of customer systems have now been restored and returned to service. We are continuing to work directly with a small number of customers whose systems require additional recovery, validation, or rebuilding. For those remaining environments, our engineers are reviewing all available backups, storage snapshots, virtual disk copies, and recovery sources. Each system is being handled individually to ensure that any restored data is complete, consistent, and safe to return to production. Our security investigation and infrastructure review are also continuing in parallel with the recovery process. Based on the evidence reviewed to date, we have found no indication that customer data was copied or removed from the environment. We will continue to validate this conclusion as the investigation progresses. We have completed several major remediation steps, including: • Rebuilding affected management infrastructure • Resetting credentials and reviewing privileged access • Removing unauthorized access and associated tools • Restoring systems from clean recovery points where possible • Reviewing network, storage, virtualization, and security logs • Strengthening segmentation and access controls across the environment Our immediate priority remains restoring the final affected systems and supporting the customers impacted by those remaining recovery cases. We understand the disruption and concern this incident has caused. We are deeply sorry for the impact on our customers and partners, and we appreciate the patience and support we have received while our team continues this work. We will provide another update as additional recovery milestones are completed and more information becomes available.

22. resolved Jul 23, 2026, 07:44 PM UTC

    Our team has completed the recovery efforts related to the ransomware incident that affected a portion of our hosted environment. All customer environments have now been restored to operational status. We have continued to validate the restored systems to confirm they are stable, complete, and safe to return to production. In parallel, our security investigation and infrastructure review remain ongoing. Based on the evidence reviewed to date, we have found no indication that customer data was copied or removed from the environment. We have completed several major remediation steps, including rebuilding the affected management infrastructure, resetting credentials and reviewing privileged access, restoring systems from clean recovery points where possible, reviewing network, storage, virtualization, and security logs, and strengthening segmentation and access controls across the environment. With the recovery process now complete, our focus has shifted to completing the formal Root Cause Analysis and Reason for Outage report, as well as implementing the improvements identified during our response. We will continue documenting our findings and completing the remaining remediation items before publishing the official report. We sincerely apologize for the disruption this incident caused. We appreciate the patience, understanding, and support we received from our partners and customers throughout the recovery process.

23. postmortem Aug 27, 2026, 04:07 PM UTC

    # **Root Cause Analysis Report** PHX Datacenter Ransomware Outage & Infrastructure Recovery ## **Executive Summary** On July 10, 2026, a widespread outage impacted the PHX Datacenter environment, affecting customer virtual machines, storage arrays, and network connectivity. Initial indicators suggested a complex multi-layered hardware, power, or storage failure. However, as troubleshooting progressed, telemetry revealed suspicious activity on a dedicated storage/management server. Subsequent forensic log analysis established that the initial unauthorized access and compromise occurred on July 9, 2026. Prompt containment actions initiated on July 10 successfully neutralized and eliminated the threat actor's foothold within the network before the end of the day. Out of an abundance of caution, the engineering team isolated the storage network and initiated a disaster recovery protocol. Over 150 TB of data was successfully restored and validated from pre-incident point-in-time snapshots. Comprehensive log reviews and outbound traffic analyses identified no evidence of customer data exfiltration. Recovery efforts, including storage restoration, virtual network rebuilding, and credential rotation, were completed by July 23, 2026. ## **Timeline of Events \(Key Milestones\)** · **July 9, 2026:** Forensic log analysis identified this date as the initial point of compromise, when unauthorized access was established via an external entry point. · **July 10, 05:34 CDT:** Widespread outage reported across PHX Datacenter and virtual machines; investigation initiated with datacenter engineers. · **July 10, 10:36 CDT:** Identification of storage array issues; initiation of recovery and snapshot mounting procedures. · **July 10, 15:40 CDT:** Discovery of indicators consistent with a potential compromise on a dedicated storage/management server; machine isolated. · **July 10, 16:34 CDT:** President issues partner notification; storage network taken offline; disaster recovery operations begin. The threat actor's active foothold and unauthorized access vectors were fully eliminated from the network by the close of the day. · **July 11, 05:14 CDT:** Approximately 150 customer VMs successfully restored and verified from July 9 and July 10 backups; network reconstruction begins. · **July 12, 10:56 – 20:39 CDT:** Physical networking restored; teams manually rebuild virtual networking/WAN configurations. · **July 14, 16:14 CDT:** Official confirmation that an unauthorized third party deployed ransomware; outbound traffic analysis identifies no evidence of data exfiltration · **July 15 – July 21:** Ongoing individual recovery efforts for remaining edge-case customer environments, credential resets, and log reviews. · **July 23, 14:44 CDT:** Final customer environments restored; system stability validated; transition to formal post-incident review and remediation. ## **Root Cause Analysis** · **Primary Cause & Ingress Date:** An unauthorized third party used compromised credentials to gain access to an externally accessible remote access service within the PHX Datacenter environment on July 9, 2026. · **Attack Progression & Containment:** The threat actor leveraged the initial access vector to deploy ransomware on July 10, impacting underlying management servers and a portion of the hosted virtual infrastructure. Swift intervention neutralized the active threat and eliminated their foothold within the network by July 10. · **Contributing Factor \(Interconnected Layers\):** Interconnected storage, virtualization, and management layers allowed the disruptive activity to mimic widespread infrastructure failure initially. · **Contributing Factor \(External Access\):** External accessibility of the entry point permitted the initial intrusion. This exposure has since been remediated \(see Corrective and Preventive Actions below\). \(Note: Specific technical entry points are withheld from this report as a security precaution\). ## **Impact Assessment** · **Service Availability:** Total loss of customer virtual machine access and environment management capabilities beginning July 10, with phased restoration concluding on July 23. · **Data Integrity:** Restored environments were rebuilt from clean backups and storage snapshots validated as predating encryption activity \(primarily July 9 and early July 10\). · **Data Privacy / Confidentiality:** Based on an extensive review of outbound network traffic, throughput limits, and security logs, no evidence was identified that customer data was copied or exfiltrated from the environment. ## **Corrective & Preventive Actions \(CAPA\)** To prevent a recurrence and enhance overall infrastructure resilience, the following remediation steps have been executed or are underway: · **Environment Rebuilding:** Rebuilt all affected management infrastructure and virtual environments from trusted, pre-incident snapshots rather than attempting in-place cleaning. · **Access Control & Credential Resets:** Conducted a comprehensive audit of privileged access, reviewed administrative permissions, and performed a total credential reset across infrastructure, storage, and virtualization platforms. · **Exfiltration Forensic Verification:** Completed a comprehensive forensic review of historical network logs, deep analysis of traffic patterns from the incident window, and cross-device access audits to verify definitively that no data exfiltration occurred. · **External Attack Surface Audit:** Completed a comprehensive audit of all external IP addresses and internet-facing services across the environment; unnecessary external exposure was eliminated, and remaining external touchpoints were validated and hardened. · **Snapshot & Backup Coverage:** Extended point-in-time snapshot coverage to all storage platforms, including tiers previously outside the snapshot schedule, ensuring complete recoverability across the environment. · **Network Segmentation:** Strengthened network segmentation and access controls between external touchpoints, management planes, and production storage fabrics. · **Enhanced Monitoring:** Expanded log aggregation and security monitoring across storage, virtualization, and network layers to improve early detection of anomalous management activity. · **Real-Time Network Behavior & Exfiltration Monitoring:** Deployed additional security controls for real-time network traffic monitoring and anomaly detection to instantly identify malicious behavioral patterns, lateral movement, or potential data exfiltration attempts in the future.

Read the full incident report →

Started Mar 30, 2026, 07:24 PM UTC· resolved May 01, 2026, 03:31 AM UTC

1. investigating Mar 30, 2026, 07:24 PM UTC

   We have received an automated alert from our monitoring system notifying us of an issue reaching certain Baremetal servers. We are reviewing this alert and are engaging with the datacenter team.

2. investigating Mar 30, 2026, 07:48 PM UTC

   A ticket was opened with the datacenter team after we confirmed we could not access the affected bare metal servers. We are currently awaiting an engineer to be assigned to this ticket to troubleshoot the issue.

3. investigating Mar 30, 2026, 08:15 PM UTC

   A datacenter engineer has been assigned to the ticket and he is currently reviewing and troubleshooting the issue.

4. identified Mar 30, 2026, 08:47 PM UTC

   The datacenter engineer has determined that the issue is network related and has escalated the ticket to the datacenter network engineering team. A network engineer has been assigned and is currently collecting network traffic logs for the affected bare metal servers. Once the collection of logs is completed, he will be reviewing the logs as well as real time traffic in order to identify the underlying cause.

5. identified Mar 30, 2026, 09:17 PM UTC

   The network engineer has confirmed that traffic flow is currently disrupted on the affected baremetal servers. Incoming, as well as outgoing traffic is currently being dropped. He is continuing to investigate to find the root cause and has engaged an additional team member from the datacenter networking team to get this issue resolved.

6. identified Mar 30, 2026, 10:09 PM UTC

   We are continuing to work with the datacenter networking team to resolve this issue. The team is actively troubleshooting, and we are currently awaiting confirmation and an estimated time to resolution. We are also on a live call with an engineer to review their findings and analyze the available data.

7. identified Mar 30, 2026, 11:10 PM UTC

   We are continuing to troubleshoot with the datacenter network engineering team. We concluded a call with them approximately 15 minutes ago, and they are actively working to resolve the issue. We expect an update from their troubleshooting efforts within the next 30 minutes.

8. identified Mar 31, 2026, 12:00 AM UTC

   We have spoken with the datacenter network engineering team again and are actively working to escalate the case for deeper investigation. The engineering team is currently performing detailed log analysis across the networking stack and has already initiated resets on key networking components as part of their troubleshooting process. Initial observations indicate that packets are successfully entering the network and egressing from upstream interfaces; however, traffic appears to be dropped or not properly forwarded at an intermediate point in the path. The team is analyzing potential causes, including routing inconsistencies, firewall or ACL enforcement, and possible issues within the switching fabric. At this time, the team is continuing to trace packet flow across the network path to identify the exact point of failure and determine root cause. We will provide further updates as additional data becomes available.

9. identified Mar 31, 2026, 01:58 AM UTC

   We apologize for the delay in communication as we have been actively working to escalate this issue. The case has now been escalated, and a senior network engineer is engaged and reviewing the situation. Recent remediation efforts and repair attempts have not yet resolved the problem. The engineering team continues to perform in-depth analysis to identify the root cause and determine the appropriate corrective actions. We will continue to provide updates as more information becomes available.

10. identified Mar 31, 2026, 11:34 AM UTC

    We understand the impact this issue is having and sincerely appreciate your patience. We are actively reaching out to the assigned engineer and their management team to obtain further clarification and a clearer understanding of the current status and next steps.

11. identified May 01, 2026, 03:31 AM UTC

    We are continuing to work on a fix for this issue.

12. resolved May 01, 2026, 03:31 AM UTC

    This incident has been resolved.

Read the full incident report →

Started Mar 17, 2026, 04:44 PM UTC· resolved Mar 20, 2026, 07:21 PM UTC

1. investigating Mar 17, 2026, 04:44 PM UTC

   We are receiving reports of some clients on Baremetal servers experiencing connectivity issues. We are working with the datacenter engineers at this time to isolate and resolve this as quickly as possible.

2. investigating Mar 17, 2026, 05:30 PM UTC

   We are continuing to investigate this issue alongside the datacenter engineers. We have confirmed that three separate bare-metal servers are currently impacted by the connectivity issue and that the issue does not appear to be caused by any hardware located on the servers themselves. Additional updates will follow as soon as they become available.

3. investigating Mar 17, 2026, 06:26 PM UTC

   Our investigation into this issue is still ongoing. Our support ticket regarding this issue has been escalated to Tier 2 of the datacenter networking team and we are actively working with them in order to resolve the ongoing connectivity issue affecting three bare metal servers. The datacenter team is currently attempting to identify the exact cause of this outage. Once we have confirmation on the exact cause of the outage and the next steps, this incident will be updated with those details.

4. investigating Mar 17, 2026, 07:42 PM UTC

   The tier 2 datacenter networking team is still working on identifying and resolving the root cause affecting the three bare metal servers. At the time of writing, we do not yet have an exact root cause. Once the exact cause has been identified, this incident will be updated with the relevant details and the steps taken to resolve the issue.

5. investigating Mar 17, 2026, 09:03 PM UTC

   Our engineering team is still currently investigating a significant issue affecting three of our bare metal environments located in Chicago, Ashburn, and Phoenix. At this time, all three locations are experiencing downtime. Initial findings indicate this may be part of a broader issue affecting multiple accounts, likely related to how intermediary virtual traffic tagging is handled across data centers. We recently completed a call with our Tier 2 engineering team and they are now working closely with the data center cloud team to conduct a deeper analysis and move toward implementing a resolution as quickly as possible. At this time, our colocation and other services remain operational and unaffected. However, we are observing an impact to network connectivity between our colocation environment and the affected bare metal datacenters, which is contributing to the overall disruption.

6. identified Mar 17, 2026, 09:59 PM UTC

   We have just spoken with the engineer currently assigned to our ticket at the data center. The issue is being actively worked on in close coordination with the data center cloud team. At this stage, the data center team is performing packet captures and investigating the behavior across multiple locations. The next step in their process is to closely review and troubleshoot the packets being routed out of our bare metal environments. Based on what is known so far, there is a working theory that the packets may contain malformed headers, which could be preventing them from routing correctly across the virtualized LAN layer. This appears to be a complex issue, and the teams involved are carefully analyzing the situation to better understand the root cause. In parallel, the data center team is also opening a support ticket with the hardware vendor on their end to bring in additional expertise and help accelerate the resolution.

7. identified Mar 17, 2026, 10:47 PM UTC

   We have received an update from the data center engineering team indicating that they are close to completing the collection of troubleshooting data for one affected client, after which our environment will be next in line for analysis. They will be gathering detailed data from our systems to support further investigation into the issue. At this time, there is no estimated timeline for resolution. The teams are focused on collecting and analyzing the necessary information to better understand the exact nature of the incident and to evaluate the most effective path forward.

8. identified Mar 18, 2026, 12:21 AM UTC

   We have received confirmation that data collection for our environment has now been completed. The engineering teams are currently reviewing the collected data and analyzing the packet captures in detail. At the same time, they are continuing to gather information from one additional affected client to help build a more complete picture of the issue. At this stage, one of the working theories is that this may be related to an internal system automation that could be contributing to the problem. This remains one of two primary areas of focus as the investigation continues. In parallel, a support ticket has been opened with the vendor responsible for routing and switching, and the matter is being actively escalated to ensure additional expertise is brought in.

9. resolved Mar 30, 2026, 07:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 09, 2026, 04:39 PM UTC· resolved Mar 09, 2026, 10:55 PM UTC

1. investigating Mar 09, 2026, 04:39 PM UTC

   We are receiving reports of an outage at our ELK Datacenter. We are working with Datacenter Engineers to isolate the cause of the issue and get it resolved as quickly as possible. We will update this status as information becomes available.

2. investigating Mar 09, 2026, 05:03 PM UTC

   We are continuing to work with datacenter engineers at this time to identify and isolate the cause of the issue. We will provide updates as they become available.

3. investigating Mar 09, 2026, 05:20 PM UTC

   We are currently coordinating with the datacenter team to troubleshoot the issue. We were able to gain access through out-of-band management and are investigating the cause of the WAN side remaining down. We are still actively working on resolving the issue and will provide updates as we learn more.

4. monitoring Mar 09, 2026, 05:43 PM UTC

   We have identified the networking issue in cooperation with the datacenter and have implemented a fix for this. We are seeing connectivity being restored at this time and expect everything to be back online within the next few minutes.

5. resolved Mar 17, 2026, 10:55 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 22, 2026, 10:24 PM UTC· resolved Feb 24, 2026, 06:28 PM UTC

1. investigating Feb 22, 2026, 10:24 PM UTC

   We are receiving reports of an outage at our PHX location. This appears to be isolated to certain VM's and we are investigating this internally and with the datacenter. We will provide updates as information becomes available.

2. identified Feb 22, 2026, 10:32 PM UTC

   We have identified the issue to be related to one of our hosts inside of the PHX datacenter. We initiated a reboot and the server is now back online. We will begin the process of rebooting all of the VM's in a staggered manner in order to make sure all of the servers come up in the proper order for Horizon functionality. This could take some time and we will update this status page once the process has been completed.

3. monitoring Feb 22, 2026, 10:57 PM UTC

   At this time, the servers have been rebooted and are returning to normal. We are continuing to monitor for ongoing issues. If you continue to experience issues, please submit a ticket to [email protected].

4. resolved Mar 09, 2026, 05:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2026, 12:43 AM UTC· resolved Feb 20, 2026, 06:28 PM UTC

1. investigating Feb 19, 2026, 12:43 AM UTC

   We are currently investigating reports of issues at our ELK datacenter. We are working with datacenter engineers to identify and remedy this issue as quickly as possible.

2. identified Feb 19, 2026, 12:47 AM UTC

   The issue appears to be isolated to an outage with networking. We are continuing to work with datacenter engineers to resolve this issue as quickly as possible.

3. monitoring Feb 19, 2026, 01:26 AM UTC

   we are seeing services being restored at this time. Preliminary indicators point to a routing failover at the datacenter which took time to propagate. We are working with the engineers at the datacenter to prevent issues like this from occurring in the future. If you continue to experience issues please email support to open a ticket.

4. resolved Mar 09, 2026, 05:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 05, 2026, 07:09 PM UTC· resolved Feb 06, 2026, 06:28 PM UTC

1. investigating Feb 05, 2026, 07:09 PM UTC

   We are receiving reports from our monitoring that a a subset of clients are currently experiencing an outage at our PHX datacenter. This issue is being investigated and we are working with the Datacenter engineers to get this resolved as quickly as possible.

2. identified Feb 05, 2026, 07:15 PM UTC

   We have identified the issue to be related to one of our hosts inside of the PHX datacenter. We initiated a reboot and the server is coming back online as we speak. We will begin the process of rebooting all of the VM's in a staggered manner in order to make sure all of the servers come up in the proper order for Horizon functionality. This could take some time and we will update this status page once the process has been completed.

3. monitoring Feb 05, 2026, 08:41 PM UTC

   At this time, the servers have been returned to normal, and we are continuing to monitor for ongoing issues. If you continue to experience issues, please submit a ticket to [email protected].

4. resolved Mar 09, 2026, 05:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 31, 2026, 04:46 PM UTC· resolved Feb 02, 2026, 06:27 PM UTC

1. investigating Jan 31, 2026, 04:46 PM UTC

   We have received reports about an outage at our ELK Datacenter. We are currently investigating this issue and working with Datacenter Engineers to resolve this problem as quickly as possible. We will update this status as information becomes available.

2. investigating Jan 31, 2026, 05:11 PM UTC

   we are continuing to investigate the issue. The primary causes at this time are believed to be a hardware or routing issue. We are interfacing with Datacenter engineers to get this resolved as quickly as possible.

3. investigating Jan 31, 2026, 05:47 PM UTC

   Datacenter engineers are still investigating this issue. We will provide more updates as information becomes available.

4. investigating Jan 31, 2026, 06:40 PM UTC

   We are continuing to investigate this issue and believe we have identified the issue in cooperation with the Datacenter engineers and are working on applying a fix.

5. investigating Jan 31, 2026, 07:29 PM UTC

   We have identified the issue as a data center core routing issue affecting our cabinets and are tracing the issue to identify the exact cause and to restore our network configuration

6. investigating Jan 31, 2026, 08:11 PM UTC

   We have begun doing a test restore of a backup of the networking configuration to new hardware

7. monitoring Jan 31, 2026, 08:30 PM UTC

   We are now seeing systems come back online after the restore and are awaiting Datacenter engineers to provide a full write up of the cause and why this took so long to be resolved. We are continuing to monitor the situation.

8. resolved Mar 09, 2026, 05:27 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 26, 2026, 09:26 PM UTC· resolved Jan 28, 2026, 06:30 PM UTC

1. investigating Jan 26, 2026, 09:26 PM UTC

   We got alerts about some of our Bare Metal Cloud servers losing network connectivity. We are investigating the alert with our datacenter team.

2. investigating Jan 26, 2026, 09:42 PM UTC

   We received alerts that all networking is back online. Outage lasted about 25 minutes; we are still identifying the root cause of this issue. This issue only affected Bare Metal Services from our datacenter. Colocation, storage and all other service remained fully operational.

3. resolved Mar 09, 2026, 05:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 15, 2026, 05:15 PM UTC· resolved Jan 16, 2026, 06:27 PM UTC

1. investigating Jan 15, 2026, 05:15 PM UTC

   We have been made aware of a detected outage affecting our ELK DC. Our engineers are currently working with engineers at the datacenter to isolate and remediate the issue. We will update this post as we gather information.

2. monitoring Jan 15, 2026, 05:33 PM UTC

   At this time all network connectivity at the ELK datacenter has been restored. Our investigation has revealed that our primary firewall at the ELK DC crashed and network traffic failed over to our secondary firewall. Upon further review, we determined that we are unable to access the primary firewall and have scheduled a firewall replacement with the datacenter team to occur after hours at 2:00 AM CST on 1/16/2026. This replacement will not impact network traffic in any way. We are continuing to monitor this situation.

3. resolved Mar 09, 2026, 05:27 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 14, 2025, 12:29 AM UTC· resolved Nov 14, 2025, 06:25 PM UTC

1. investigating Nov 14, 2025, 12:29 AM UTC

   We have received reports that there is an ongoing networking issue at our ELK datacenter. We are currently engaged with datacenter engineers to get this problem resolved as soon as possible. We will update this status as information becomes available.

2. monitoring Nov 14, 2025, 12:53 AM UTC

   The issue has been identified and resolved and we are continuing to monitor the situation at this time. If any further updates are needed we will provide them at that time.

3. resolved Mar 09, 2026, 05:25 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 29, 2025, 10:11 PM UTC· resolved Oct 31, 2025, 05:22 PM UTC

1. monitoring Oct 29, 2025, 10:11 PM UTC

   at 5 PM Central Time we received alerts from our automated monitoring that there was a networking blip that occurred at our ELK Datacenter. At this time it appears that all services have been restored and are back to a stable state. We will continue to monitor this and provide updates as needed.

2. resolved Mar 09, 2026, 05:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 08, 2025, 01:11 PM UTC· resolved Jul 09, 2025, 05:30 PM UTC

1. investigating Jul 08, 2025, 01:11 PM UTC

   We are currently aware of ongoing intermittent issues with connectivity to some Horizon instances at our PHX Datacenter. We are actively working to identify the issue and resolve it as quickly as possible with the Datacenter. We will be updating this post with any new information we gather.

2. monitoring Jul 08, 2025, 01:56 PM UTC

   The intermittent issues related to connectivity that we identified earlier have all subsided as of 8:13 AM CST and have not re-occurred. The intermittent issue only affected a small group of tenant environments. At this time, we have not identified any underlying issues on our infrastructure that would have caused this temporary network blip but will continue to monitor the connection.

3. resolved Mar 09, 2026, 05:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 18, 2025, 05:25 PM UTC· resolved Jun 20, 2025, 05:21 PM UTC

1. monitoring Jun 18, 2025, 05:25 PM UTC

   We received reports in our monitoring about packet loss to our PHX Datacenter. Some clients may have experienced an interruption to their connection, however, this interruption lasted less than 2 minutes and no lasting impact is expected. We will continue to monitor as always.

2. resolved Mar 09, 2026, 05:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 28, 2025, 04:14 PM UTC· resolved May 29, 2025, 02:47 PM UTC

1. investigating May 28, 2025, 04:14 PM UTC

   We have received reports of VM's becoming unresponsive and are working to identify and resolve this issue with datacenter engineers as quickly as possible. We will be posting updates as we get them.

2. identified May 28, 2025, 04:30 PM UTC

   We have identified the issue and are working with the datacenter to restore networking services.

3. identified May 28, 2025, 04:41 PM UTC

   We are seeing networking being restored at this time and are waiting for everything to finish coming back to begin verifying functionality.

4. identified May 28, 2025, 05:30 PM UTC

   We are actively investigating this issue. Currently, we’re experiencing difficulties re-establishing communication between the hosts and storage devices. Our team is working closely with the datacenter to resolve this as swiftly as possible.

5. identified May 28, 2025, 06:45 PM UTC

   The datacenter team has resolved all outstanding issues on their end. We have also resolved an additional issue and are in the process of bringing the affected virtual machines back online. As virtual machines come back online and are confirmed to be operational, we will be issuing another status update.

6. identified May 28, 2025, 07:45 PM UTC

   All services are in the process of being restored. Virtual machines should be coming up and a portion of them are already up. Networking issues have all be resolved, part of this issue was internal networking between management and production clusters and our storage network. The team is engaged on bringing up all servers in the proper order and confirming logins.

7. monitoring May 28, 2025, 09:26 PM UTC

   At this time, all services appear to have been restored and we have received no further reports of ongoing issues. If you or your clients are still continuing to experience any issues, please submit a support ticket or reach out to us via phone to receive support on a case by case basis.

8. resolved May 29, 2025, 02:47 PM UTC

   We have continued to monitor the environment through the evening into this morning and have confirmed that no further issues are occurring. We will be marking this incident as resolved at this time. We expect to provide a postmortem and additional details in regards to this outage in the next two weeks.

Read the full incident report →

Started Apr 14, 2025, 10:30 PM UTC· resolved Apr 14, 2025, 10:45 PM UTC

1. monitoring Apr 14, 2025, 10:30 PM UTC

   We received reports from our monitoring about a brief networking blip that lasted approximately 3 minutes at our ELK datacenter. At this time all traffic is flowing normally as expected and we will be continuing to monitor it.

2. resolved May 29, 2025, 02:39 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 10, 2025, 03:45 PM UTC· resolved Mar 10, 2025, 05:38 PM UTC

1. investigating Mar 10, 2025, 03:45 PM UTC

   We are aware of an ongoing network outage at our Elk Datacenter. We are currently in the processing of investigating. We believe this to be a failover event. We will update this status once we gather more information

2. identified Mar 10, 2025, 04:03 PM UTC

   ELK is experiencing a partial network outage at the moment; we are aware of the issue and are actively working on getting this resolved.

3. identified Mar 10, 2025, 04:28 PM UTC

   Our data center engineering team is currently onsite at our rack in the ELK data center, actively investigating a network issue. A technician is working to restore connectivity and determine why the WAN failover did not activate as expected. At this time, approximately 30% of clients at ELK are affected. We are treating this as a priority and will provide further updates as more information becomes available.

4. identified Mar 10, 2025, 05:14 PM UTC

   Our data center engineering team has brought the network back online and virtual machines should be acquiring WAN access automatically right now. We are still investigating the cause of this outage.

5. monitoring Mar 10, 2025, 05:26 PM UTC

   WAN networking has been fully restored. Our team is actively investigating the root cause of this issue, including why the automated failover did not function as expected. We will provide a follow-up update once our investigation into today’s outage is complete.

6. resolved May 29, 2025, 02:38 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 22, 2025, 07:09 PM UTC· resolved Mar 10, 2025, 04:01 PM UTC

1. investigating Jan 22, 2025, 07:09 PM UTC

   We have detected a network outage affecting the ELK datacenter. We are currently investigating this issue.

2. identified Jan 22, 2025, 07:10 PM UTC

   We have detected that a piece of networking hardware failed-over and are investigating the cause of this fail-over event.

3. monitoring Jan 22, 2025, 10:22 PM UTC

   We have determined the issue to be caused by an upstream routing fail-over event and have coordinated with the datacenter to see how we can prevent this from happening in the future. Our monitoring detected approximately 3 min of down time due to this event.

4. resolved Mar 10, 2025, 04:01 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 06, 2025, 05:22 PM UTC· resolved Mar 10, 2025, 04:01 PM UTC

1. investigating Jan 06, 2025, 05:22 PM UTC

   We are investigating an outage on one of our management hosts affecting multiple VM's.

2. identified Jan 06, 2025, 05:32 PM UTC

   We have identified one of the Management hosts experienced a PSOD. The host is back up now and virtual machines are being booted back up. We have not yet identified the cause of the PSOD but will be investigating it once all the virtual machines are back to normal operation.

3. monitoring Jan 06, 2025, 06:09 PM UTC

   All virtual machines are back online and operational. We have migrated workloads to other hosts while we investigate the PSOD on the affected host.

4. monitoring Mar 10, 2025, 04:01 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Mar 10, 2025, 04:01 PM UTC

   This incident has been resolved.

Read the full incident report →