postmortem Feb 17, 2026, 05:48 PM UTC
On February 16, 2026, an incident identified as INC-262 occurred, resulting in a critical network outage that affected all systems in our Frankfurt data centers for a duration of 8 minutes. The incident was triggered by a routine configuration change made to a static route on one of our routers. Prior to implementation, the configuration underwent validation via the operating system \(Juniper Junos OS\), which generally checks for errors. However, due to a bug in the firmware, the system incorrectly accepted a faulty configuration change. 4 minutes after the change was applied, both forwarding engines were reported unreachable by the routing engine. The issue was promptly detected and reported by our monitoring systems. An automatic rollback occurred one minute following the failure due to a lack of manual confirmation, which restored both routing engines and forwarding engines. Safety mechanisms have worked as expected and resulted in a quick recovery completed by 7:27 PM. To preemptively harden our infrastructure against similar issues in the future, we already planned to transition from a redundant routing chassis to completely isolated devices, ensuring full redundancy even in case of such failures. This transition is scheduled for April/May during a planned maintenance. We appreciate your understanding and patience as we work to enhance our systems' reliability.