Traffic delayed and increased errors
Affected: Traffic Processing
Redox Outage History
Redox is up right nowRedox had 48 outages in the last 2 years totaling 91h 14m of downtime — averaging 2 incidents per month.
There were 48 Redox outages since August 25, 2024 totaling 91h 14m of downtime. Each is summarised below — incident details, duration, and resolution information.
Customer configured alerts not fully functional
Affected: Dashboard Tools
Timeline · 3 updates
- investigating May 28, 2025, 05:32 PM UTC
A secondary system we use for our Customer Configured Alerts is currently experiencing an outage. This may result in missed alerts or some extra alerts. Being sent out. This only affects the Customer Configured Alerts not all our other alerting systems. We are monitoring the impact of this secondary system outage.
- monitoring May 28, 2025, 06:26 PM UTC
The secondary system's incident is resolved. All Customer Configured Alerts are back up and running. We are investigating possible impact for what alerts may not have been correctly emailed during the outage. You can look at the Monitoring page on your dashboard to see an accurate current status of all customer configured alerts.
- resolved May 28, 2025, 07:51 PM UTC
Alerting continues to function as expected. We are reaching out to customers who may have missed emails who are still in an alerted state individually.
Delay in Email Alerting
Affected: Platform API
Timeline · 3 updates
- identified May 14, 2025, 10:50 PM UTC
We are aware of an issue with out alerting system that has delayed email alerting due to connection issues by up to two hours. Our development team is implementing a fix, and Redox is actively monitoring the situation.
- monitoring May 14, 2025, 11:14 PM UTC
We've implemented a fix and are presently monitoring for further impact
- resolved May 15, 2025, 02:23 AM UTC
This incident has been resolved.
Viewing details in some Logs may be delayed
Affected: Dashboard Tools
Timeline · 3 updates
- investigating May 12, 2025, 09:14 PM UTC
We're currently observing a delay impacting the visibility of the transmission information in some logs. There are NO delays in actual message traffic or delivery — all data is transmitting as expected. Our team is actively working to reduce the latency and will keep you updated on any significant changes. Thank you for your understanding and patience.
- monitoring May 12, 2025, 10:09 PM UTC
We have identified the issue and have implemented a change. Log visibility is no longer delayed. We are continuing to monitor the issue as we work on deploying a long term fix.
- resolved May 12, 2025, 11:32 PM UTC
This incident has been resolved.
Customer log content may be delayed
Affected: Dashboard Tools
Timeline · 5 updates
- investigating May 09, 2025, 05:02 PM UTC
We're currently observing a delay affecting the archive importing system, which impacts the visibility of message logs and transmission contents. To clarify: there are no delays in actual message traffic or delivery — all data is transmitting as expected. However, some customers may experience up to a 5-minute delay in viewing these logs within the system. Our team is actively working to reduce the latency and will keep you updated on any significant changes. Thank you for your understanding and patience.
- investigating May 09, 2025, 05:39 PM UTC
We are continuing to investigate this issue.
- investigating May 09, 2025, 05:39 PM UTC
We are continuing to investigate this issue.
- monitoring May 09, 2025, 05:40 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 09, 2025, 08:18 PM UTC
This incident has been resolved.
Editing filters is currently unavailable in the dashboard
Affected: Dashboard Tools
Timeline · 2 updates
- identified Apr 08, 2025, 12:46 PM UTC
Editing filters is currently unavailable in the dashboard. New filters can still be created, and already created filters are still working as intended. Already created filters cannot be edited. The issue has been identified and a fix is being implemented.
- resolved Apr 08, 2025, 01:27 PM UTC
This incident has been resolved.
Some email notification services are down
Affected: Dashboard Tools
Timeline · 4 updates
- identified Feb 13, 2025, 05:04 PM UTC
Two email notification services that Redox uses are down. These services affect emails for notifying customers of expiring certificates and for message traffic or VPN related issues. We have identified the issue and are implementing a fix. When the issue is resolved, customers will receive all backlogged emails that would have been sent while the services were down.
- monitoring Feb 13, 2025, 05:15 PM UTC
A fix has been implemented and customers will now start receiving the queued emails. We are continuing to monitor the results.
- resolved Feb 13, 2025, 05:38 PM UTC
This incident has been resolved.
- postmortem Feb 20, 2025, 09:29 PM UTC
## Summary On February 11th at 11:30 AM CT, a small number\(<20\) email notifications to users were delayed. These notifications were not sent until February 13th at 12:35 PM CT. ## What Happened * On February 11th, a misconfigured API key in a new notification service prevented email notifications from being sent from a subset of systems. * As a result, customers experienced delays in receiving: * Notifications about traffic errors and resolutions. * Notifications about upcoming certificate expirations \(30 days out\). * The API key was updated and normal processing resumed on February 13th at 12:35 PM CT. ## What we are doing about this: * Additional monitoring and alerting were added to prevent similar potential failures in the future.
Filters are unavailable in the dashboard UI. Processing is not impacted.
Affected: Dashboard Tools
Timeline · 3 updates
High latency in transmission processing for some customers
Affected: Traffic Processing
Timeline · 6 updates
- investigating Feb 03, 2025, 11:32 PM UTC
We are aware that some customers may be experiencing increased latency in transmission processing. We are actively looking into this issue, and will update as we know more information
- investigating Feb 04, 2025, 12:43 AM UTC
We are aware that some of the affected customers are also receiving NACKs on their transmissions. We are continuing to investigate the root cause of this issue.
- identified Feb 04, 2025, 05:06 AM UTC
We have identified the root cause of the issue causing the message latency and are in the process of finding a fix. If you have any questions, please reach out to [email protected]
- monitoring Feb 04, 2025, 06:10 AM UTC
We have implemented a fix that is mitigating the effects of the latency we've been seeing. We are actively monitoring to ensure that this fix holds
- monitoring Feb 04, 2025, 07:18 AM UTC
We are continuing to monitor the fix that we've implemented. Almost all queues (fewer than a dozen remaining) have drained, and the rest are set to drain by midday today (February 4th). If you have any questions, feel free to reach out to [email protected]
- resolved Feb 04, 2025, 12:52 PM UTC
This incident has been resolved.
Log Processing Delay for a Subset of Customers
Affected: Traffic Processing
Timeline · 3 updates
- investigating Jan 29, 2025, 03:27 PM UTC
We are investigating an issue that may delay asynchronous traffic log processing for a subset of users.
- monitoring Jan 29, 2025, 03:37 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 29, 2025, 04:24 PM UTC
This incident has been resolved.
Slow traffic and increased latency
Affected: Traffic ProcessingPlatform API
Timeline · 4 updates
Additional Customer Filter Errors
Affected: Dashboard Tools
Timeline · 4 updates
Customer Filter Errors
Affected: Dashboard Tools
Timeline · 4 updates
Message Processing Halted for Subset of Customers
Affected: Traffic Processing
Timeline · 4 updates
- investigating Dec 13, 2024, 01:00 PM UTC
Redox is investigating an issue affecting message processing for a subset of customers. Affected customers may notice a stop to inbound traffic from Redox and an inability to send messages outbound to Redox.
- identified Dec 13, 2024, 01:07 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Dec 13, 2024, 01:13 PM UTC
A fix has been implemented and we are monitoring the results. Customers should see that traffic flow has resumed and may notice a larger queue depth until caught up.
- resolved Dec 13, 2024, 01:29 PM UTC
This incident has been resolved.
Advanced MD credentials locked out
Timeline · 2 updates
- identified Dec 04, 2024, 10:09 PM UTC
The issue has been identified and a fix has been implemented. We are now working with the appropriate AdvancedMD teams to have our credentials reset so polling and other workflows can be resumed.
- resolved Dec 05, 2024, 01:16 AM UTC
A large majority of polling workflows have been re-enabled and traffic should be flowing as expected. A small subset of credentials do need to be reset but we have communicated with these customers individually to resolve these on a case by case basis. The incident is now considered resolved.
Carequality directory modification requests are failing
Affected: Carequality
Timeline · 4 updates
- investigating Nov 22, 2024, 05:29 PM UTC
This appears to be a broader Carequality issue at this time and we have reached out to Carequality support to help identify and troubleshoot these request errors.
- investigating Nov 22, 2024, 10:59 PM UTC
We understand the impacts of this issue and our teams are continuing to collaborate with Carequality and internal partners. Your understanding and patience is appreciated while we continue to work toward a resolution.
- monitoring Nov 22, 2024, 11:54 PM UTC
A fix has been implemented. We have updated our end to meet the updated Carequality standard and messages are going through as expected. We will continue to monitor on our end for any additional unexpected changes.
- resolved Nov 23, 2024, 12:17 AM UTC
This incident has been resolved.
Up to half hour delay for a small subset of customers
Timeline · 2 updates
- monitoring Nov 21, 2024, 10:27 PM UTC
Connections where we poll external APIs for messages will be delayed by up to 30 minutes. The issue has been Identified and a fix has been released. We are not monitoring.
- resolved Nov 21, 2024, 10:46 PM UTC
This incident has been resolved.
Log Visibility Unavailable
Affected: Dashboard Tools
Timeline · 4 updates
- investigating Nov 20, 2024, 03:48 PM UTC
Redox is investigating an issue affecting the ability to view Logs in the Dashboard. Log processing is NOT affected. Messages are processing as expected. The visibility of logs in the dashboard is unavailable.
- identified Nov 20, 2024, 03:56 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Nov 20, 2024, 04:01 PM UTC
This incident has been resolved. Log visibility is available.
- postmortem Nov 26, 2024, 03:14 PM UTC
## Summary * On November 20, 2024 from 0915CT to 1000CT Logs details were not visible in the dashboard due to logs processing delays. Message processing was not affected. ## What Happened * We made a code change that changed the underlying infrastructure for Logs. * We had planned for this type of failure and as a result were quickly able to fail forward. ## What are we doing about it? * We will expand our internal alerting and monitoring to include an endpoint that would have enabled us to identify this type of issue more quickly.
Dashboard Search Functionality Limited
Affected: Dashboard Tools
Timeline · 5 updates
- investigating Oct 28, 2024, 09:54 PM UTC
We are aware that dashboard search functionality is extremely limited at the moment. Present workaround is not to use the 'all' category when searching, but we are actively working on a fix.
- identified Oct 28, 2024, 10:18 PM UTC
We have identified the issue and are working on a solution presently
- monitoring Oct 28, 2024, 11:21 PM UTC
We've applied a fix in production and are actively monitoring log searches to ensure that full functionality has been restored
- resolved Oct 29, 2024, 07:31 AM UTC
This incident has been resolved.
- postmortem Nov 12, 2024, 04:00 PM UTC
## Summary * From October 25-28, 2024 some Logs searches initiated by use of the search bar errored. ## What Happened * On October 25 we implemented a code change, which was tested in isolation. * On October 28 we were alerted to search errors and started an incident at 1647CT. * At 1710CT we identified the issue and started working on a fix. * At 1819CT the fix was fully deployed and functionality in Logs search was restored. ## What are we doing about it? * We are considering expanding our testing capabilities to account for earlier detection and mitigation of this type of failure in the future.
Message processing is delayed for a subset of customers
Affected: Platform API
Timeline · 3 updates
- investigating Oct 01, 2024, 03:51 PM UTC
Message processing is delayed for a subset of customers. Observability is not impacted. We are currently investigating the issue.
- identified Oct 01, 2024, 03:59 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Oct 01, 2024, 04:04 PM UTC
A fix has been implemented and processing has returned to normal. This incident has been resolved.
Message Processing Degraded for a Subset of Customers
Affected: Dashboard Tools
Timeline · 2 updates
- investigating Sep 16, 2024, 12:15 PM UTC
Some customers may experience a delay in message processing while we investigate this issue. Thank you for your patience.
- resolved Sep 16, 2024, 12:38 PM UTC
This incident has been resolved.
Dashboard logs are delayed at this time. Message processing is not impacted.
Affected: Dashboard Tools
Timeline · 8 updates
Dashboard logs are up to two hours behind. Processing should not be impacted.
Affected: Dashboard Tools