Incident multiple POPs affected

Update timeline

1. resolved Mar 12, 2025, 12:26 AM UTC

   On March 1, 2025, Link11 monitoring systems detected packet drops across multiple Points of Presence (POPs). The incident was promptly investigated by the on-shift Security Operations Center (SOC) personnel and escalated through the operational hierarchy to the Chief Technology Officer (CTO). The root cause was identified as an unexpected interaction between Link11's filtering software and the network driver when handling a spike of traffic composed of uncommon IP protocols. A workaround was successfully implemented, restoring normal operations within approximately four hours. Affected POPs: ams1 – Amsterdam, Netherlands lax1 – Los Angeles, United States mia1 – Miami, United States nyc1 – New York City, United States ffm7 – Frankfurt, Germany ffm14 – Frankfurt, Germany lon2 – London, United Kingdom sgp1 – Singapore, Singapore ash3 – Ashburn, United States stl1 – St. Louis, United States Affected services: Infrastructure DDoS Protection Web DDoS Protection Bot Protection Zero Touch WAF

2. postmortem Mar 12, 2025, 12:26 AM UTC

   **Root Cause:** The incident was triggered by an unusual spike in traffic consisting of uncommon IP protocols. This traffic pattern caused an unforeseen interaction between Link11's filtering software and its network driver, leading to packet drops. The filtering appliance struggled to process these packets efficiently, resulting in service degradation across multiple POPs. **Mitigation** he initial proposed solution was to deploy a temporary workaround to ensure stability while a permanent fix will be developed. The Link11 Dev-Team tracks this under DDOSF-413 to ensure continuous stability and resilience of the Link11’s filtering software. As a continuous action we are refining escalation and response procedures to minimize time-to-resolution for similar incidents.` ` **Conclusion:** This incident highlights the importance of proactive anomaly detection and robust filtering mechanisms to handle unexpected traffic spikes. The Link11 SOC and engineering teams have been escalating this matter swiftly, using all available resources to minimize disruption and implement a workaround to restore normal service level in a timely manner. Moving forward, continuous improvements in filtering software and escalation protocols will further enhance Link11’s resilience against such incidents.