Is Redox down?

We are aware of an issue where logs are not visible in the dashboard. While logs are still processing properly, the dashboard is not displaying them as expected. If you have questions about specific logs or your connection and whether it has been affected, please contact us at [email protected]

We have identified the cause for the log visibility issues and are actively working on a fix. We will update the status of this incident when the fix has been deployed.

We have pushed a fix that will make new logs visible, however logs received between approximately 5:45PM and 8:40AM will still not be viewable. The backlogs of logs will continue to process through the queue while it catches up.

A fix has been implemented and visibility for the logs in the backlog is returning. We are actively monitoring the situation and ensuring that all affected logs are visible again.

This incident has been resolved. Log visibility is restored. If you have any questions, please reach out to [email protected]

## Summary Following an emergency failover carried out in response to a third-party outage on September 8, a configuration change made to stabilize that failover infrastructure reduced capacity on the system powering dashboard and log visibility. Message processing itself was not affected — data continued to be sent and received normally throughout this incident. Customers began experiencing delayed visibility into message logs and payload details overnight; we identified the issue by 7:45 AM Central on September 9 and fully resolved it by 1:51 PM Central. No data was lost at any point. ## What Happened As part of stabilizing the backup infrastructure used to restore service during a separate third-party outage earlier that day, our team identified an opportunity to make its scaling configuration more resilient to future failover deployments, and moved quickly to put that improvement in place. That change was deployed around 4:34 PM Central on September 8. It did not have the intended effect, and capacity on the backup system supporting dashboard and log visibility was reduced as a result. Message processing itself was not affected. Monitoring coverage on that temporary infrastructure was still being extended at the time, so the reduced capacity was not immediately flagged, and a visibility backlog accumulated overnight. ## Impact Customers experienced delayed visibility into message logs and payload details in the Customer Dashboard, beginning overnight and continuing into the morning of September 9. Message processing was not affected — data continued to be sent and received normally. The impact was limited to the dashboard and log visibility layer. ## How We Resolved It Customer reports and internal alerts led our team to begin investigating at approximately 7:30 AM Central on September 9. Within 15 minutes, we identified that the failover infrastructure was underscaled and began remediation. Between 8:00 AM and 12:00 PM Central, we: * Increased infrastructure capacity to add processing headroom * Scaled the services powering dashboard and log visibility * Distributed traffic across infrastructure to accelerate recovery Visibility into message logs returned to fully caught-up status by 12:51 PM Central. We returned to normal operating capacity by 1:51 PM Central and confirmed resolution on our status page. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * Require monitoring and alerting to be fully wired up on any temporary or backup infrastructure before a team stands down from an incident * Add automated deployment tests to catch configuration resets before they reach production, including on temporary or failover infrastructure * Consider implementing absolute minimum scale limits so services cannot drop below production-safe thresholds, while preserving a standard, well-documented method for intentionally adjusting capacity when needed We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _This issue arose while our team was stabilizing infrastructure used in an emergency failover for a separate third-party outage. See \[Incident 1: Third-Party Service Outage & Processing Delay\] for details._

Our automated monitoring tools detected a major outage affecting the API and main site/dashboard at 1:34pm Central. Our team is currently investigating the cause and working on a solution. Please work with your teams to implement downtime procedures. If you have any additional questions, please notify us at [email protected].

We applied a fix at 2:23pm Central and processing appears to be operating as normal. We are still seeing delays in log visibility. We will continue to monitor to verify that the issues are fully resolved with our 3rd party vendor.

The incident has been resolved. Please do not hesitate to reach out to [email protected] if you are continuing to run into anything unexpected.

## Summary On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost. ## What Happened In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing. This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved. ## Impact Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours. ## How We Resolved It Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * We are actively migrating away from this third-party provider We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See \[Incident 2: Dashboard & Log Visibility Delay\] for details._

At approximately 11:43am CT, Redox became aware of an issue with DocumentGet requests sent outbound to other organizations on the Carequality network. What this means: DocumentGet requests to retrieve documents from other organizations on the Carequality network might not return the document to the initiator. What we're doing: We are currently investigating and will provide updates as they become available. Please contact us at [email protected] if you have questions.

A fix has been implemented for and document requests are now working as expected. We're actively monitoring the situation to ensure continued successes. If you had failed DocumentGet queries for Carequality you can re-query and you will now get responses. If you have any additional questions, please contact [email protected]

The incident has been resolved and the Redox Engine has resumed normal operations. If you have any additional questions please reach out to [email protected]

## Summary Between Wednesday, July 8, 2026 at approximately 1:00 PM CT and Thursday, July 9, 2026 at 12:47 PM CT, customers using Carequality DocumentGet received empty document data in responses. Requests appeared to complete successfully on both ends, but document content was missing from what was returned. PatientSearch and PatientQuery were not affected. Service has been fully restored, and no data was lost. All documents were successfully retrieved during this window but not correctly included in outbound responses. ## What Happened The incident was triggered by a routine software update deployed on Wednesday afternoon. * The update introduced an incompatibility in how we format and map document data into outbound responses. Rather than producing an error, the affected step silently returned empty content. * Because the system reported no errors and appeared healthy, our automated monitoring did not alert. The issue was identified through customer reports on Thursday morning. * Once our engineering team identified the root cause, we rolled back the update, which restored correct behavior. Service was confirmed recovered at 12:47 PM CT on July 9. ## What We Are Doing About This To prevent this from happening again and to catch similar issues faster, we are taking the following actions: * **Fixing the Underlying Incompatibility:** We are updating the specific configuration affected by this change so the original update can be safely re-applied without reintroducing the issue. * **Broader Configuration Audit:** We are reviewing other configurations that could be affected by the same underlying change to ensure no similar failures exist elsewhere in the platform. * **Improved Monitoring:** We are adding monitoring that validates response content — not just whether a request completed — so that data-omission failures trigger alerts rather than going undetected. * **Expanded Test Coverage:** We are adding tests that verify the actual content of mapped responses, so this class of failure is caught during development before it reaches production.

The RedoxEngine message/transmission logs have fallen behind starting at 10:15 AM Central. Our team is currently investigating the cause and working on a solution. Overall message traffic is unaffected. If you have any additional questions, please notify us at [email protected].

We are continuing to investigate this issue.

We have identified the cause for the log visibility issues and are actively working on a fix. We are seeing the number of messages with delays decrease. We will update the status of this incident when said fix has been deployed.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.