Baremetal Alert

Update timeline

1. investigating Mar 30, 2026, 07:24 PM UTC

   We have received an automated alert from our monitoring system notifying us of an issue reaching certain Baremetal servers. We are reviewing this alert and are engaging with the datacenter team.

2. investigating Mar 30, 2026, 07:48 PM UTC

   A ticket was opened with the datacenter team after we confirmed we could not access the affected bare metal servers. We are currently awaiting an engineer to be assigned to this ticket to troubleshoot the issue.

3. investigating Mar 30, 2026, 08:15 PM UTC

   A datacenter engineer has been assigned to the ticket and he is currently reviewing and troubleshooting the issue.

4. identified Mar 30, 2026, 08:47 PM UTC

   The datacenter engineer has determined that the issue is network related and has escalated the ticket to the datacenter network engineering team. A network engineer has been assigned and is currently collecting network traffic logs for the affected bare metal servers. Once the collection of logs is completed, he will be reviewing the logs as well as real time traffic in order to identify the underlying cause.

5. identified Mar 30, 2026, 09:17 PM UTC

   The network engineer has confirmed that traffic flow is currently disrupted on the affected baremetal servers. Incoming, as well as outgoing traffic is currently being dropped. He is continuing to investigate to find the root cause and has engaged an additional team member from the datacenter networking team to get this issue resolved.

6. identified Mar 30, 2026, 10:09 PM UTC

   We are continuing to work with the datacenter networking team to resolve this issue. The team is actively troubleshooting, and we are currently awaiting confirmation and an estimated time to resolution. We are also on a live call with an engineer to review their findings and analyze the available data.

7. identified Mar 30, 2026, 11:10 PM UTC

   We are continuing to troubleshoot with the datacenter network engineering team. We concluded a call with them approximately 15 minutes ago, and they are actively working to resolve the issue. We expect an update from their troubleshooting efforts within the next 30 minutes.

8. identified Mar 31, 2026, 12:00 AM UTC

   We have spoken with the datacenter network engineering team again and are actively working to escalate the case for deeper investigation. The engineering team is currently performing detailed log analysis across the networking stack and has already initiated resets on key networking components as part of their troubleshooting process. Initial observations indicate that packets are successfully entering the network and egressing from upstream interfaces; however, traffic appears to be dropped or not properly forwarded at an intermediate point in the path. The team is analyzing potential causes, including routing inconsistencies, firewall or ACL enforcement, and possible issues within the switching fabric. At this time, the team is continuing to trace packet flow across the network path to identify the exact point of failure and determine root cause. We will provide further updates as additional data becomes available.

9. identified Mar 31, 2026, 01:58 AM UTC

   We apologize for the delay in communication as we have been actively working to escalate this issue. The case has now been escalated, and a senior network engineer is engaged and reviewing the situation. Recent remediation efforts and repair attempts have not yet resolved the problem. The engineering team continues to perform in-depth analysis to identify the root cause and determine the appropriate corrective actions. We will continue to provide updates as more information becomes available.

10. identified Mar 31, 2026, 11:34 AM UTC

    We understand the impact this issue is having and sincerely appreciate your patience. We are actively reaching out to the assigned engineer and their management team to obtain further clarification and a clearer understanding of the current status and next steps.

11. identified May 01, 2026, 03:31 AM UTC

    We are continuing to work on a fix for this issue.

12. resolved May 01, 2026, 03:31 AM UTC

    This incident has been resolved.