Adeptcore Outage History

The datacenter engineer has determined that the issue is network related and has escalated the ticket to the datacenter network engineering team. A network engineer has been assigned and is currently collecting network traffic logs for the affected bare metal servers. Once the collection of logs is completed, he will be reviewing the logs as well as real time traffic in order to identify the underlying cause.

The network engineer has confirmed that traffic flow is currently disrupted on the affected baremetal servers. Incoming, as well as outgoing traffic is currently being dropped. He is continuing to investigate to find the root cause and has engaged an additional team member from the datacenter networking team to get this issue resolved.

We are continuing to work with the datacenter networking team to resolve this issue. The team is actively troubleshooting, and we are currently awaiting confirmation and an estimated time to resolution. We are also on a live call with an engineer to review their findings and analyze the available data.

We are continuing to troubleshoot with the datacenter network engineering team. We concluded a call with them approximately 15 minutes ago, and they are actively working to resolve the issue. We expect an update from their troubleshooting efforts within the next 30 minutes.

We have spoken with the datacenter network engineering team again and are actively working to escalate the case for deeper investigation. The engineering team is currently performing detailed log analysis across the networking stack and has already initiated resets on key networking components as part of their troubleshooting process. Initial observations indicate that packets are successfully entering the network and egressing from upstream interfaces; however, traffic appears to be dropped or not properly forwarded at an intermediate point in the path. The team is analyzing potential causes, including routing inconsistencies, firewall or ACL enforcement, and possible issues within the switching fabric. At this time, the team is continuing to trace packet flow across the network path to identify the exact point of failure and determine root cause. We will provide further updates as additional data becomes available.

We apologize for the delay in communication as we have been actively working to escalate this issue. The case has now been escalated, and a senior network engineer is engaged and reviewing the situation. Recent remediation efforts and repair attempts have not yet resolved the problem. The engineering team continues to perform in-depth analysis to identify the root cause and determine the appropriate corrective actions. We will continue to provide updates as more information becomes available.

We understand the impact this issue is having and sincerely appreciate your patience. We are actively reaching out to the assigned engineer and their management team to obtain further clarification and a clearer understanding of the current status and next steps.

We are continuing to work on a fix for this issue.

We have just spoken with the engineer currently assigned to our ticket at the data center. The issue is being actively worked on in close coordination with the data center cloud team. At this stage, the data center team is performing packet captures and investigating the behavior across multiple locations. The next step in their process is to closely review and troubleshoot the packets being routed out of our bare metal environments. Based on what is known so far, there is a working theory that the packets may contain malformed headers, which could be preventing them from routing correctly across the virtualized LAN layer. This appears to be a complex issue, and the teams involved are carefully analyzing the situation to better understand the root cause. In parallel, the data center team is also opening a support ticket with the hardware vendor on their end to bring in additional expertise and help accelerate the resolution.

We have received an update from the data center engineering team indicating that they are close to completing the collection of troubleshooting data for one affected client, after which our environment will be next in line for analysis. They will be gathering detailed data from our systems to support further investigation into the issue. At this time, there is no estimated timeline for resolution. The teams are focused on collecting and analyzing the necessary information to better understand the exact nature of the incident and to evaluate the most effective path forward.

We have received confirmation that data collection for our environment has now been completed. The engineering teams are currently reviewing the collected data and analyzing the packet captures in detail. At the same time, they are continuing to gather information from one additional affected client to help build a more complete picture of the issue. At this stage, one of the working theories is that this may be related to an internal system automation that could be contributing to the problem. This remains one of two primary areas of focus as the investigation continues. In parallel, a support ticket has been opened with the vendor responsible for routing and switching, and the matter is being actively escalated to ensure additional expertise is brought in.

We have identified the issue to be related to one of our hosts inside of the PHX datacenter. We initiated a reboot and the server is now back online. We will begin the process of rebooting all of the VM's in a staggered manner in order to make sure all of the servers come up in the proper order for Horizon functionality. This could take some time and we will update this status page once the process has been completed.

The issue appears to be isolated to an outage with networking. We are continuing to work with datacenter engineers to resolve this issue as quickly as possible.

We have identified the issue to be related to one of our hosts inside of the PHX datacenter. We initiated a reboot and the server is coming back online as we speak. We will begin the process of rebooting all of the VM's in a staggered manner in order to make sure all of the servers come up in the proper order for Horizon functionality. This could take some time and we will update this status page once the process has been completed.