Is Adeptcore down?

We have received an automated alert from our monitoring system notifying us of an issue reaching certain Baremetal servers. We are reviewing this alert and are engaging with the datacenter team.

A ticket was opened with the datacenter team after we confirmed we could not access the affected bare metal servers. We are currently awaiting an engineer to be assigned to this ticket to troubleshoot the issue.

A datacenter engineer has been assigned to the ticket and he is currently reviewing and troubleshooting the issue.

The datacenter engineer has determined that the issue is network related and has escalated the ticket to the datacenter network engineering team. A network engineer has been assigned and is currently collecting network traffic logs for the affected bare metal servers. Once the collection of logs is completed, he will be reviewing the logs as well as real time traffic in order to identify the underlying cause.

The network engineer has confirmed that traffic flow is currently disrupted on the affected baremetal servers. Incoming, as well as outgoing traffic is currently being dropped. He is continuing to investigate to find the root cause and has engaged an additional team member from the datacenter networking team to get this issue resolved.

We are continuing to work with the datacenter networking team to resolve this issue. The team is actively troubleshooting, and we are currently awaiting confirmation and an estimated time to resolution. We are also on a live call with an engineer to review their findings and analyze the available data.

We are continuing to troubleshoot with the datacenter network engineering team. We concluded a call with them approximately 15 minutes ago, and they are actively working to resolve the issue. We expect an update from their troubleshooting efforts within the next 30 minutes.

We have spoken with the datacenter network engineering team again and are actively working to escalate the case for deeper investigation. The engineering team is currently performing detailed log analysis across the networking stack and has already initiated resets on key networking components as part of their troubleshooting process. Initial observations indicate that packets are successfully entering the network and egressing from upstream interfaces; however, traffic appears to be dropped or not properly forwarded at an intermediate point in the path. The team is analyzing potential causes, including routing inconsistencies, firewall or ACL enforcement, and possible issues within the switching fabric. At this time, the team is continuing to trace packet flow across the network path to identify the exact point of failure and determine root cause. We will provide further updates as additional data becomes available.

We apologize for the delay in communication as we have been actively working to escalate this issue. The case has now been escalated, and a senior network engineer is engaged and reviewing the situation. Recent remediation efforts and repair attempts have not yet resolved the problem. The engineering team continues to perform in-depth analysis to identify the root cause and determine the appropriate corrective actions. We will continue to provide updates as more information becomes available.

We understand the impact this issue is having and sincerely appreciate your patience. We are actively reaching out to the assigned engineer and their management team to obtain further clarification and a clearer understanding of the current status and next steps.

We are continuing to work on a fix for this issue.

This incident has been resolved.

We are receiving reports of some clients on Baremetal servers experiencing connectivity issues. We are working with the datacenter engineers at this time to isolate and resolve this as quickly as possible.

We are continuing to investigate this issue alongside the datacenter engineers. We have confirmed that three separate bare-metal servers are currently impacted by the connectivity issue and that the issue does not appear to be caused by any hardware located on the servers themselves. Additional updates will follow as soon as they become available.

Our investigation into this issue is still ongoing. Our support ticket regarding this issue has been escalated to Tier 2 of the datacenter networking team and we are actively working with them in order to resolve the ongoing connectivity issue affecting three bare metal servers. The datacenter team is currently attempting to identify the exact cause of this outage. Once we have confirmation on the exact cause of the outage and the next steps, this incident will be updated with those details.

The tier 2 datacenter networking team is still working on identifying and resolving the root cause affecting the three bare metal servers. At the time of writing, we do not yet have an exact root cause. Once the exact cause has been identified, this incident will be updated with the relevant details and the steps taken to resolve the issue.

Our engineering team is still currently investigating a significant issue affecting three of our bare metal environments located in Chicago, Ashburn, and Phoenix. At this time, all three locations are experiencing downtime. Initial findings indicate this may be part of a broader issue affecting multiple accounts, likely related to how intermediary virtual traffic tagging is handled across data centers. We recently completed a call with our Tier 2 engineering team and they are now working closely with the data center cloud team to conduct a deeper analysis and move toward implementing a resolution as quickly as possible. At this time, our colocation and other services remain operational and unaffected. However, we are observing an impact to network connectivity between our colocation environment and the affected bare metal datacenters, which is contributing to the overall disruption.

We have just spoken with the engineer currently assigned to our ticket at the data center. The issue is being actively worked on in close coordination with the data center cloud team. At this stage, the data center team is performing packet captures and investigating the behavior across multiple locations. The next step in their process is to closely review and troubleshoot the packets being routed out of our bare metal environments. Based on what is known so far, there is a working theory that the packets may contain malformed headers, which could be preventing them from routing correctly across the virtualized LAN layer. This appears to be a complex issue, and the teams involved are carefully analyzing the situation to better understand the root cause. In parallel, the data center team is also opening a support ticket with the hardware vendor on their end to bring in additional expertise and help accelerate the resolution.

We have received an update from the data center engineering team indicating that they are close to completing the collection of troubleshooting data for one affected client, after which our environment will be next in line for analysis. They will be gathering detailed data from our systems to support further investigation into the issue. At this time, there is no estimated timeline for resolution. The teams are focused on collecting and analyzing the necessary information to better understand the exact nature of the incident and to evaluate the most effective path forward.

We have received confirmation that data collection for our environment has now been completed. The engineering teams are currently reviewing the collected data and analyzing the packet captures in detail. At the same time, they are continuing to gather information from one additional affected client to help build a more complete picture of the issue. At this stage, one of the working theories is that this may be related to an internal system automation that could be contributing to the problem. This remains one of two primary areas of focus as the investigation continues. In parallel, a support ticket has been opened with the vendor responsible for routing and switching, and the matter is being actively escalated to ensure additional expertise is brought in.

This incident has been resolved.