Temporary Redundancy Degradation at firstcolo FRA4

Update timeline

1. investigating Jul 23, 2025, 01:27 PM UTC

   We are currently investigating intermittent port flaps between our Edge routing (Interxion FRA8) and our location firstcolo FRA4. These events may temporarily reduce redundancy and cause increased latency. At this time, we have not observed any packet loss, but we are working to stabilize the situation promptly to ensure continued reliability, especially during peak hours. While we are troubleshooting, there is an increased risk of service impact at this datacenter. We will provide an update as soon as more details are available.

2. identified Jul 23, 2025, 04:05 PM UTC

   The situation remains stable, with no packet loss observed, but we continue to see increases in latency. We currently suspect a faulty transceiver as the root cause. Since multiple components are involved (routers, switches, converters, DWDM equipment) across two different datacenters, identifying and replacing the defective part requires a step-by-step process and therefore takes time. Our teams are working in parallel at both datacenters to resolve this as quickly as possible. To further improve redundancy and capacity, we already ordered additional dark fibers and opened a new PoP at Equinix FR4 a few weeks ago in response to bandwidth growth. These links have a lead time of approximately 8–10 weeks. Once they are in service, a failure on a single line will no longer cause any impact at all, as intended.

3. resolved Jul 23, 2025, 05:02 PM UTC

   After migrating the transport service to alternative interfaces on the muxponders in both datacenters, the situation has stabilized. Latencies have decreased and are no longer spiking. Since the devices did not report any errors, we had to replace nearly all optical transceivers involved before taking this final (and resolving) action. The issue has now been resolved for about 15 minutes without any recurrence.