Redox experienced a major incident on May 6, 2026 affecting Traffic Processing, lasting 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 06, 2026, 10:20 PM UTC
At approximately 4:45pm CT, Redox became aware of an issue impacting VPN tunnels. Impacted tunnels will see errors in attempts to deliver messages to destinations. What this means: you may experience a delay before receiving traffic from Redox. We are actively looking into this issue and will have a fix available as soon as possible. If you are concerned that your logs have been impacted or have any additional questions, please reach out to [email protected].
- identified May 06, 2026, 10:38 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 06, 2026, 10:55 PM UTC
A fix has been implemented and we are monitoring the results. VPN message traffic is resuming as normal.
- resolved May 06, 2026, 11:14 PM UTC
Messages are now processing normally. While we were unable to send VPN traffic we queued messages and now that VPNs are up and stable we are sending out all those messages. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]
- postmortem May 22, 2026, 09:56 PM UTC
### Summary Between **4:20 PM CT and 5:38 PM CT** on May 6, 2026, some customers experienced an interruption in outbound traffic for VPN-based connections. During this window, data egressing via the network proxy service failed to authorize, resulting in unsuccessful transmissions for MLLP traffic. Service was fully restored by **6:10 PM CT** after configuration settings were corrected and gateway services were restarted. ### What Happened The service interruption was triggered during a planned migration of our network configuration management system. * **Summary of Causes:** During the transition to a new configuration framework, an authentication credential used by the network proxy service was incorrectly identified as unused and subsequently removed. While pre-deployment testing appeared successful, the removal of this credential caused authorization failures once applied across the broader production environment. * **Summary of Impact:** VPN-based connections for outbound data experienced a loss of egress connectivity. They would get an error on the first outbound message on a subscription to a VPN destination. Transmissions were queued during the issue and sent after the problem was resolved; no transmissions were dropped. ### What we are doing about this: * **Enhanced Testing Protocols:** We are developing new end-to-end tests specifically for outbound MLLP traffic to ensure that connectivity is verified across all gateways during configuration changes. * **Improved Service Observability:** We are updating our alerting logic for our proxy services to trigger more immediate notifications if egress traffic volume drops below expected thresholds, ensuring faster detection.