xByte Cloud incident
Monitoring Alert Unhealthy - Connections being lost
xByte Cloud experienced a notice incident on November 3, 2024 affecting Zone C - US Central, lasting 2h 47m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 03, 2024, 10:22 PM UTC
xByte Cloud is investigating a network connection issue that is impacting connection requests from a specific location. Further details will be provided as soon as they are available.
- investigating Nov 03, 2024, 10:23 PM UTC
The xByte Cloud network monitors are currently showing all systems are healthy. We’re reaching out to our upstream ISP providers to investigate any specific connection routing issues.
- investigating Nov 03, 2024, 10:59 PM UTC
We’re still observing connection losses in transit to our network, and our network team is actively continuing the investigation.
- investigating Nov 03, 2024, 11:32 PM UTC
Our network team is still observing connection losses on approximately 50% of our inbound requests. We are continuing to investigate the issue beyond our network.
- investigating Nov 04, 2024, 12:13 AM UTC
The service issue persists, affecting specific subnets for inbound requests. We are still assessing the incident and are not yet able to provide confirmed details.
- investigating Nov 04, 2024, 12:26 AM UTC
Our network team reports that traffic from previously affected subnets is now successfully reaching our network and establishing connections.
- monitoring Nov 04, 2024, 12:44 AM UTC
Our team will be reaching out to affected customers to confirm that functionality is as expected, as we are currently not seeing any impacted subnets. We are still reviewing the incident and awaiting the release of a full post-mortem report.
- resolved Nov 04, 2024, 01:06 AM UTC
We are still reviewing the incident and awaiting the release of a full post-mortem report.
- postmortem Nov 04, 2024, 05:11 PM UTC
**Incident Overview:** On November 3rd, at approximately 4:00 PM CST, our ISP experienced a network disruption that impacted a subset of customer traffic. This incident temporarily prevented certain external sources from reaching specific destinations within our network. While most services continued to operate normally, certain destinations were unreachable from some external sources. **Cause of the Incident:** The issue was traced to a switch within our ISP’s network infrastructure that handles Layer2 traffic for part of our services. Layer2 ports responsible for directing traffic to specific endpoints experienced forwarding issues, which impacted connectivity. Once the affected switch was isolated and traffic rerouted to an alternative path, services began to stabilize. **Resolution Steps:** Our ISP promptly took the impacted switch out of service, conducted diagnostic checks, and applied updates to address the issue and prevent similar incidents in the future. After thorough testing, the switch was restored to service, and stable operations have resumed. **Impact on Services:** This incident may have affected connectivity for customers whose traffic relied on the impacted switch paths. We apologize for any inconvenience and appreciate your patience as we worked with our ISP to restore full functionality. We are committed to ensuring a robust and resilient network and continue to collaborate with our ISP to minimize future risks.