Redox incident
High latency in transmission processing for some customers
Redox experienced a minor incident on February 3, 2025 affecting Traffic Processing, lasting 13h 20m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 03, 2025, 11:32 PM UTC
We are aware that some customers may be experiencing increased latency in transmission processing. We are actively looking into this issue, and will update as we know more information
- investigating Feb 04, 2025, 12:43 AM UTC
We are aware that some of the affected customers are also receiving NACKs on their transmissions. We are continuing to investigate the root cause of this issue.
- identified Feb 04, 2025, 05:06 AM UTC
We have identified the root cause of the issue causing the message latency and are in the process of finding a fix. If you have any questions, please reach out to [email protected]
- monitoring Feb 04, 2025, 06:10 AM UTC
We have implemented a fix that is mitigating the effects of the latency we've been seeing. We are actively monitoring to ensure that this fix holds
- monitoring Feb 04, 2025, 07:18 AM UTC
We are continuing to monitor the fix that we've implemented. Almost all queues (fewer than a dozen remaining) have drained, and the rest are set to drain by midday today (February 4th). If you have any questions, feel free to reach out to [email protected]
- resolved Feb 04, 2025, 12:52 PM UTC
This incident has been resolved.