Redox Outage History

**What Happened** On **February 26, 2026** at approximately **10:16 AM CT**, our production environment experienced elevated latency across all asynchronous traffic processing. The incident was triggered when core processing workers were unintentionally scaled to a minimal scale, below our normal expected scale to process day to day traffic. This significantly reduced our processing capacity. The impact varied across connections, with durations ranging from 20-42 minutes and maximum average latencies between 9.6 and 18.2 minutes. **How We Fixed It** Our team quickly identified the root cause and executed a script to scale all affected workers back to their proper scale. By **10:45 AM CT**, all async processing had returned to nominal latency levels.

### **Summary** On **February 23, 2026**, between **1:05 AM to 2:05 AM CST** and **1:53 PM to 3:03 PM CST**, a subset of customers experienced elevated latency and connectivity issues across a number of our MLLP VPN Gateways. Service was fully restored once primary gateway instances were cycled and traffic successfully failed over to secondary nodes. ### **What Happened** The outage was triggered by a memory exhaustion event on specific VPN Gateway instances. There was a memory leak in a third party library led to a steady rise in memory consumption, eventually causing the primary instances to become unresponsive. ### **What We Are Doing** To prevent a recurrence and improve our response time, our engineering team is implementing the following measures: * **Stability:** We are deploying updates that mitigate the issue we saw with the third party memory leak to prevent resource exhaustion and non-responsive primary gateway instances. We submitted a ticket with the third party library and they have resolved the issue in the library. * **Enhanced Monitoring:** We are deploying updates to have more granular alerts that monitor MLLP gateway traffic health and reduce time to detection for unhealthy gateways. * **Service Resilience:** We are reviewing our automated health monitoring and failover protocols to ensure the system responds effectively to instance health changes.

### **Summary** On **February 23, 2026**, between **1:05 AM to 2:05 AM CST** and **1:53 PM to 3:03 PM CST**, a subset of customers experienced elevated latency and connectivity issues across a number of our MLLP VPN Gateways. Service was fully restored once primary gateway instances were cycled and traffic successfully failed over to secondary nodes. ### **What Happened** The outage was triggered by a memory exhaustion event on specific VPN Gateway instances. There was a memory leak in a third party library led to a steady rise in memory consumption, eventually causing the primary instances to become unresponsive. ### **What We Are Doing** To prevent a recurrence and improve our response time, our engineering team is implementing the following measures: * **Stability:** We are deploying updates that mitigate the issue we saw with the third party memory leak to prevent resource exhaustion and non-responsive primary gateway instances. We submitted a ticket with the third party library and they have resolved the issue in the library. * **Enhanced Monitoring:** We are deploying updates to have more granular alerts that monitor MLLP gateway traffic health and reduce time to detection for unhealthy gateways. * **Service Resilience:** We are reviewing our automated health monitoring and failover protocols to ensure the system responds effectively to instance health changes.