Redox incident

Documents are not returned in Carequality DocumentGet requests

Minor Resolved View vendor source →

Redox experienced a minor incident on July 9, 2026 affecting Carequality, lasting 1h 17m. The incident has been resolved; the full update timeline is below.

Started
Jul 09, 2026, 04:59 PM UTC
Resolved
Jul 09, 2026, 06:16 PM UTC
Duration
1h 17m
Detected by Pingoru
Jul 09, 2026, 04:59 PM UTC

Affected components

Carequality

Update timeline

  1. investigating Jul 09, 2026, 04:59 PM UTC

    At approximately 11:43am CT, Redox became aware of an issue with DocumentGet requests sent outbound to other organizations on the Carequality network. What this means: DocumentGet requests to retrieve documents from other organizations on the Carequality network might not return the document to the initiator. What we're doing: We are currently investigating and will provide updates as they become available. Please contact us at [email protected] if you have questions.

  2. monitoring Jul 09, 2026, 05:32 PM UTC

    A fix has been implemented for and document requests are now working as expected. We're actively monitoring the situation to ensure continued successes. If you had failed DocumentGet queries for Carequality you can re-query and you will now get responses. If you have any additional questions, please contact [email protected]

  3. resolved Jul 09, 2026, 06:16 PM UTC

    The incident has been resolved and the Redox Engine has resumed normal operations. If you have any additional questions please reach out to [email protected]

  4. postmortem Jul 15, 2026, 04:51 PM UTC

    ## Summary Between Wednesday, July 8, 2026 at approximately 1:00 PM CT and Thursday, July 9, 2026 at 12:47 PM CT, customers using Carequality DocumentGet received empty document data in responses. Requests appeared to complete successfully on both ends, but document content was missing from what was returned. PatientSearch and PatientQuery were not affected. Service has been fully restored, and no data was lost. All documents were successfully retrieved during this window but not correctly included in outbound responses. ## What Happened The incident was triggered by a routine software update deployed on Wednesday afternoon. * The update introduced an incompatibility in how we format and map document data into outbound responses. Rather than producing an error, the affected step silently returned empty content. * Because the system reported no errors and appeared healthy, our automated monitoring did not alert. The issue was identified through customer reports on Thursday morning. * Once our engineering team identified the root cause, we rolled back the update, which restored correct behavior. Service was confirmed recovered at 12:47 PM CT on July 9. ## What We Are Doing About This To prevent this from happening again and to catch similar issues faster, we are taking the following actions: * **Fixing the Underlying Incompatibility:** We are updating the specific configuration affected by this change so the original update can be safely re-applied without reintroducing the issue. * **Broader Configuration Audit:** We are reviewing other configurations that could be affected by the same underlying change to ensure no similar failures exist elsewhere in the platform. * **Improved Monitoring:** We are adding monitoring that validates response content — not just whether a request completed — so that data-omission failures trigger alerts rather than going undetected. * **Expanded Test Coverage:** We are adding tests that verify the actual content of mapped responses, so this class of failure is caught during development before it reaches production.