Issue With Message Processing

Update timeline

1. investigating Feb 23, 2026, 07:09 AM UTC

   At approximately 12:30AMCT, Redox became aware of an issue with our message processing for a subset of customers. What we will do: After a fix has been implemented, Redox will go through affected messages and determine what action can be taken to rectify the situation If you have any additional questions please reach out to us at [email protected]

2. monitoring Feb 23, 2026, 07:30 AM UTC

   A fix has been implemented for the traffic processing issue. Messages are processing properly again. If you have any questions, please contact us at [email protected]

3. resolved Feb 23, 2026, 07:51 AM UTC

   Messages are now processing normally. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]

4. postmortem Mar 02, 2026, 03:29 PM UTC

   ### **Summary** On **February 23, 2026**, between **1:05 AM to 2:05 AM CST** and **1:53 PM to 3:03 PM CST**, a subset of customers experienced elevated latency and connectivity issues across a number of our MLLP VPN Gateways. Service was fully restored once primary gateway instances were cycled and traffic successfully failed over to secondary nodes. ### **What Happened** The outage was triggered by a memory exhaustion event on specific VPN Gateway instances. There was a memory leak in a third party library led to a steady rise in memory consumption, eventually causing the primary instances to become unresponsive. ### **What We Are Doing** To prevent a recurrence and improve our response time, our engineering team is implementing the following measures: * **Stability:** We are deploying updates that mitigate the issue we saw with the third party memory leak to prevent resource exhaustion and non-responsive primary gateway instances. We submitted a ticket with the third party library and they have resolved the issue in the library. * **Enhanced Monitoring:** We are deploying updates to have more granular alerts that monitor MLLP gateway traffic health and reduce time to detection for unhealthy gateways. * **Service Resilience:** We are reviewing our automated health monitoring and failover protocols to ensure the system responds effectively to instance health changes.