Redox Outage History

Started Sep 09, 2026, 12:54 PM UTC· resolved Sep 09, 2026, 06:51 PM UTC

1. investigating Sep 09, 2026, 12:54 PM UTC

   We are aware of an issue where logs are not visible in the dashboard. While logs are still processing properly, the dashboard is not displaying them as expected. If you have questions about specific logs or your connection and whether it has been affected, please contact us at [email protected]

2. identified Sep 09, 2026, 01:21 PM UTC

   We have identified the cause for the log visibility issues and are actively working on a fix. We will update the status of this incident when the fix has been deployed.

3. identified Sep 09, 2026, 01:51 PM UTC

   We have pushed a fix that will make new logs visible, however logs received between approximately 5:45PM and 8:40AM will still not be viewable. The backlogs of logs will continue to process through the queue while it catches up.

4. monitoring Sep 09, 2026, 02:38 PM UTC

   A fix has been implemented and visibility for the logs in the backlog is returning. We are actively monitoring the situation and ensuring that all affected logs are visible again.

5. resolved Sep 09, 2026, 06:51 PM UTC

   This incident has been resolved. Log visibility is restored. If you have any questions, please reach out to [email protected]

6. postmortem Sep 14, 2026, 10:27 PM UTC

   ## Summary Following an emergency failover carried out in response to a third-party outage on September 8, a configuration change made to stabilize that failover infrastructure reduced capacity on the system powering dashboard and log visibility. Message processing itself was not affected — data continued to be sent and received normally throughout this incident. Customers began experiencing delayed visibility into message logs and payload details overnight; we identified the issue by 7:45 AM Central on September 9 and fully resolved it by 1:51 PM Central. No data was lost at any point. ## What Happened As part of stabilizing the backup infrastructure used to restore service during a separate third-party outage earlier that day, our team identified an opportunity to make its scaling configuration more resilient to future failover deployments, and moved quickly to put that improvement in place. That change was deployed around 4:34 PM Central on September 8. It did not have the intended effect, and capacity on the backup system supporting dashboard and log visibility was reduced as a result. Message processing itself was not affected. Monitoring coverage on that temporary infrastructure was still being extended at the time, so the reduced capacity was not immediately flagged, and a visibility backlog accumulated overnight. ## Impact Customers experienced delayed visibility into message logs and payload details in the Customer Dashboard, beginning overnight and continuing into the morning of September 9. Message processing was not affected — data continued to be sent and received normally. The impact was limited to the dashboard and log visibility layer. ## How We Resolved It Customer reports and internal alerts led our team to begin investigating at approximately 7:30 AM Central on September 9. Within 15 minutes, we identified that the failover infrastructure was underscaled and began remediation. Between 8:00 AM and 12:00 PM Central, we: * Increased infrastructure capacity to add processing headroom * Scaled the services powering dashboard and log visibility * Distributed traffic across infrastructure to accelerate recovery Visibility into message logs returned to fully caught-up status by 12:51 PM Central. We returned to normal operating capacity by 1:51 PM Central and confirmed resolution on our status page. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * Require monitoring and alerting to be fully wired up on any temporary or backup infrastructure before a team stands down from an incident * Add automated deployment tests to catch configuration resets before they reach production, including on temporary or failover infrastructure * Consider implementing absolute minimum scale limits so services cannot drop below production-safe thresholds, while preserving a standard, well-documented method for intentionally adjusting capacity when needed We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _This issue arose while our team was stabilizing infrastructure used in an emergency failover for a separate third-party outage. See \[Incident 1: Third-Party Service Outage & Processing Delay\] for details._

Read the full incident report →

Started Sep 08, 2026, 06:59 PM UTC· resolved Sep 08, 2026, 09:04 PM UTC

1. investigating Sep 08, 2026, 06:59 PM UTC

   Our automated monitoring tools detected a major outage affecting the API and main site/dashboard at 1:34pm Central. Our team is currently investigating the cause and working on a solution. Please work with your teams to implement downtime procedures. If you have any additional questions, please notify us at [email protected].

2. monitoring Sep 08, 2026, 07:25 PM UTC

   We applied a fix at 2:23pm Central and processing appears to be operating as normal. We are still seeing delays in log visibility. We will continue to monitor to verify that the issues are fully resolved with our 3rd party vendor.

3. resolved Sep 08, 2026, 09:04 PM UTC

   The incident has been resolved. Please do not hesitate to reach out to [email protected] if you are continuing to run into anything unexpected.

4. postmortem Sep 14, 2026, 10:23 PM UTC

   ## Summary On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost. ## What Happened In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing. This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved. ## Impact Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours. ## How We Resolved It Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * We are actively migrating away from this third-party provider We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See \[Incident 2: Dashboard & Log Visibility Delay\] for details._

Read the full incident report →

Started Jul 09, 2026, 04:59 PM UTC· resolved Jul 09, 2026, 06:16 PM UTC

1. investigating Jul 09, 2026, 04:59 PM UTC

   At approximately 11:43am CT, Redox became aware of an issue with DocumentGet requests sent outbound to other organizations on the Carequality network. What this means: DocumentGet requests to retrieve documents from other organizations on the Carequality network might not return the document to the initiator. What we're doing: We are currently investigating and will provide updates as they become available. Please contact us at [email protected] if you have questions.

2. monitoring Jul 09, 2026, 05:32 PM UTC

   A fix has been implemented for and document requests are now working as expected. We're actively monitoring the situation to ensure continued successes. If you had failed DocumentGet queries for Carequality you can re-query and you will now get responses. If you have any additional questions, please contact [email protected]

3. resolved Jul 09, 2026, 06:16 PM UTC

   The incident has been resolved and the Redox Engine has resumed normal operations. If you have any additional questions please reach out to [email protected]

4. postmortem Jul 15, 2026, 04:51 PM UTC

   ## Summary Between Wednesday, July 8, 2026 at approximately 1:00 PM CT and Thursday, July 9, 2026 at 12:47 PM CT, customers using Carequality DocumentGet received empty document data in responses. Requests appeared to complete successfully on both ends, but document content was missing from what was returned. PatientSearch and PatientQuery were not affected. Service has been fully restored, and no data was lost. All documents were successfully retrieved during this window but not correctly included in outbound responses. ## What Happened The incident was triggered by a routine software update deployed on Wednesday afternoon. * The update introduced an incompatibility in how we format and map document data into outbound responses. Rather than producing an error, the affected step silently returned empty content. * Because the system reported no errors and appeared healthy, our automated monitoring did not alert. The issue was identified through customer reports on Thursday morning. * Once our engineering team identified the root cause, we rolled back the update, which restored correct behavior. Service was confirmed recovered at 12:47 PM CT on July 9. ## What We Are Doing About This To prevent this from happening again and to catch similar issues faster, we are taking the following actions: * **Fixing the Underlying Incompatibility:** We are updating the specific configuration affected by this change so the original update can be safely re-applied without reintroducing the issue. * **Broader Configuration Audit:** We are reviewing other configurations that could be affected by the same underlying change to ensure no similar failures exist elsewhere in the platform. * **Improved Monitoring:** We are adding monitoring that validates response content — not just whether a request completed — so that data-omission failures trigger alerts rather than going undetected. * **Expanded Test Coverage:** We are adding tests that verify the actual content of mapped responses, so this class of failure is caught during development before it reaches production.

Read the full incident report →

Started Jun 22, 2026, 04:02 PM UTC· resolved Jun 23, 2026, 03:07 PM UTC

1. investigating Jun 22, 2026, 04:02 PM UTC

   The RedoxEngine message/transmission logs have fallen behind starting at 10:15 AM Central. Our team is currently investigating the cause and working on a solution. Overall message traffic is unaffected. If you have any additional questions, please notify us at [email protected].

2. investigating Jun 22, 2026, 04:27 PM UTC

   We are continuing to investigate this issue.

3. identified Jun 22, 2026, 05:59 PM UTC

   We have identified the cause for the log visibility issues and are actively working on a fix. We are seeing the number of messages with delays decrease. We will update the status of this incident when said fix has been deployed.

4. monitoring Jun 22, 2026, 06:50 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Jun 23, 2026, 03:07 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 06, 2026, 10:20 PM UTC· resolved May 06, 2026, 11:14 PM UTC

1. investigating May 06, 2026, 10:20 PM UTC

   At approximately 4:45pm CT, Redox became aware of an issue impacting VPN tunnels. Impacted tunnels will see errors in attempts to deliver messages to destinations. What this means: you may experience a delay before receiving traffic from Redox. We are actively looking into this issue and will have a fix available as soon as possible. If you are concerned that your logs have been impacted or have any additional questions, please reach out to [email protected].

2. identified May 06, 2026, 10:38 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 06, 2026, 10:55 PM UTC

   A fix has been implemented and we are monitoring the results. VPN message traffic is resuming as normal.

4. resolved May 06, 2026, 11:14 PM UTC

   Messages are now processing normally. While we were unable to send VPN traffic we queued messages and now that VPNs are up and stable we are sending out all those messages. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]

5. postmortem May 22, 2026, 09:56 PM UTC

   ### Summary Between **4:20 PM CT and 5:38 PM CT** on May 6, 2026, some customers experienced an interruption in outbound traffic for VPN-based connections. During this window, data egressing via the network proxy service failed to authorize, resulting in unsuccessful transmissions for MLLP traffic. Service was fully restored by **6:10 PM CT** after configuration settings were corrected and gateway services were restarted. ### What Happened The service interruption was triggered during a planned migration of our network configuration management system. * **Summary of Causes:** During the transition to a new configuration framework, an authentication credential used by the network proxy service was incorrectly identified as unused and subsequently removed. While pre-deployment testing appeared successful, the removal of this credential caused authorization failures once applied across the broader production environment. * **Summary of Impact:** VPN-based connections for outbound data experienced a loss of egress connectivity. They would get an error on the first outbound message on a subscription to a VPN destination. Transmissions were queued during the issue and sent after the problem was resolved; no transmissions were dropped. ### What we are doing about this: * **Enhanced Testing Protocols:** We are developing new end-to-end tests specifically for outbound MLLP traffic to ensure that connectivity is verified across all gateways during configuration changes. * **Improved Service Observability:** We are updating our alerting logic for our proxy services to trigger more immediate notifications if egress traffic volume drops below expected thresholds, ensuring faster detection.

Read the full incident report →

Started Apr 13, 2026, 04:12 PM UTC· resolved Apr 13, 2026, 09:11 PM UTC

1. investigating Apr 13, 2026, 04:12 PM UTC

   We are aware of an issue with organizations querying the Redox Gateway over the Carequality network. Queries may receive an increased rate of HTTP 500 responses. All traffic not related to Carequality is unaffected. If you have further questions regarding this issue, please reach out to [email protected]

2. resolved Apr 13, 2026, 09:11 PM UTC

   A fix has been implemented and we are monitoring the results. Error rates have returned to a normal range. If you have further questions, please reach out to [email protected]

Read the full incident report →

Started Mar 16, 2026, 02:51 PM UTC· resolved Mar 16, 2026, 03:37 PM UTC

1. investigating Mar 16, 2026, 02:51 PM UTC

   Redox is currently experiencing a delay in some logs. About an eighth of logs are delayed by under and hour. What this means: you may experience a delay before receiving traffic from Redox. However, please be aware that the actual processing of logs is unaffected; message flow is still occurring, albeit at a slower pace. We are actively looking into this issue and will have a fix available as soon as possible. If you are concerned that your logs have been impacted or have any additional questions, please reach out to [email protected]

2. monitoring Mar 16, 2026, 03:06 PM UTC

   A fix has been implemented for the delay issues and Redox is actively monitoring our logs. You may continue to experience delays as queues catch up to the present. If you have questions please reach out to [email protected]

3. resolved Mar 16, 2026, 03:37 PM UTC

   The incident has been resolved and logs delays are at nominal levels. If you have any additional questions please reach out to [email protected]

Read the full incident report →

Started Mar 05, 2026, 08:48 PM UTC· resolved Mar 10, 2026, 04:36 PM UTC

1. investigating Mar 05, 2026, 08:48 PM UTC

   Our automated monitoring tools detected a partial outage affecting the dashboard at ~2:00 PM Central. Intermittently, alert rules are unable to be loaded within the dashboard and a subset of alerts are not triggering as expected. Our team is currently investigating the cause and working on a solution. Please work with your teams to implement downtime procedures. Traffic processing is not impacted. If you have any additional questions, please notify us at [email protected].

2. identified Mar 05, 2026, 10:25 PM UTC

   We're continuing to investigate the root cause of the issue. It appears to be related to a third-party vendor. We are seeing degraded performance rather than an outage at this time. We are reaching out to this partner to assist with additional troubleshooting. Additionally, we are taking steps to put failovers in place to ensure reliability of our dashboard alerts. A fix is also being put in place to ensure the dashboard still displays as expected.

3. identified Mar 05, 2026, 11:40 PM UTC

   A fix has been put in place for the dashboard so it now reflects current alert statuses as expected. We are continuing to work toward more reliable alert notification services.

4. identified Mar 06, 2026, 02:26 PM UTC

   We have identified the cause for the ongoing notification issue and are actively implementing a fix. We will update the status of this incident when this fix has been fully deployed.

5. identified Mar 06, 2026, 04:48 PM UTC

   The fix was deployed but we are still seeing intermittent errors on our end. We have been communicating with our 3rd party vendor to assist further. We will continue to provide updates as they become available.

6. identified Mar 06, 2026, 04:49 PM UTC

   The fix was deployed but we are still seeing intermittent errors on our end. We have been communicating with our 3rd party vendor to assist further. We will continue to provide updates as they become available.

7. identified Mar 06, 2026, 09:33 PM UTC

   Our third-party vendor is in the process of recovering from an incident on their end. We are doing everything within our power on the Redox side to mitigate from our end. We will continue to provide updates as they become available and will have more information available on Monday. Alerting within the customer dashboard will be inconsistent for now and through the weekend. We recommend reviewing any alerts with scrutiny to ensure they are accurate. Our own Redox internal alerting and remains unimpacted. This includes automated destination level error retry and failure alerts.

8. identified Mar 09, 2026, 02:56 PM UTC

   Our third party vendor has completed their recovery and is monitoring things as are we. Alerts are working as expected and we closing monitoring the situation to ensure this continues. If you have any additional questions, please contact [email protected]

9. monitoring Mar 09, 2026, 02:56 PM UTC

   Our third party vendor has completed their recovery and is monitoring things as are we. Alerts are working as expected and we closing monitoring the situation to ensure this continues. If you have any additional questions, please contact [email protected]

10. resolved Mar 10, 2026, 04:36 PM UTC

    This issue has been resolved as our third party vendor has resolved the issue and our alerts stayed operational through our monitoring process. If you have any additional questions please reach out to [email protected]

Read the full incident report →

Started Feb 27, 2026, 06:14 PM UTC· resolved Feb 27, 2026, 06:26 PM UTC

1. investigating Feb 27, 2026, 06:14 PM UTC

   Redox is aware of an issue with our message processing for Data on Demand. What this means: We see ~15 min delay in processing than expected. We’ve found the source of the trouble and the fix is in place. Systems are returning to normal, and you should see full functionality restored shortly. If you have any additional questions please reach out to us at [email protected]

2. monitoring Feb 27, 2026, 06:16 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 27, 2026, 06:26 PM UTC

   Messages are now processing normally. All affected queues have been resumed. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]

Read the full incident report →

Started Feb 26, 2026, 05:33 PM UTC· resolved Feb 26, 2026, 05:47 PM UTC

1. identified Feb 26, 2026, 05:33 PM UTC

   We are aware of an issue with our message processing. What this means: Messages are presently not processing as expected. Logs are taking longer to process than expected. Please expect delays in message receipt until we have identified the issue and implemented a fix. What we will do: After a fix has been implemented, Redox will go through affected messages and determine what action can be taken to rectify the situation If you have any additional questions please reach out to us at [email protected]

2. monitoring Feb 26, 2026, 05:39 PM UTC

   A fix has been implemented for the traffic processing issue and we are monitoring the results. Message processing latency is catching up. If you have any questions, please contact us at [email protected]

3. resolved Feb 26, 2026, 05:47 PM UTC

   Messages are now processing normally. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]

4. postmortem Mar 10, 2026, 10:56 PM UTC

   **What Happened** On **February 26, 2026** at approximately **10:16 AM CT**, our production environment experienced elevated latency across all asynchronous traffic processing. The incident was triggered when core processing workers were unintentionally scaled to a minimal scale, below our normal expected scale to process day to day traffic. This significantly reduced our processing capacity. The impact varied across connections, with durations ranging from 20-42 minutes and maximum average latencies between 9.6 and 18.2 minutes. **How We Fixed It** Our team quickly identified the root cause and executed a script to scale all affected workers back to their proper scale. By **10:45 AM CT**, all async processing had returned to nominal latency levels.

Read the full incident report →

Started Feb 23, 2026, 08:31 PM UTC· resolved Feb 23, 2026, 09:14 PM UTC

1. identified Feb 23, 2026, 08:31 PM UTC

   We have identified the issue with our VPN gateways and are currently working on a fix.

2. identified Feb 23, 2026, 08:38 PM UTC

   We have identified the issue with our VPN gateways and are currently working on a fix. A subset of HL7 over VPN traffic will be impacted.

3. monitoring Feb 23, 2026, 08:55 PM UTC

   Our gateways have been restarted and VPN tunnels are coming back up. We are currently monitoring results.

4. monitoring Feb 23, 2026, 08:59 PM UTC

   As of 02/23/2026 at ~2:15 PM Central Time, the Redox MLLP Listener service is experiencing degraded performance for processing inbound MLLP transactions. Inbound and outbound HL7 messages to and from Redox were delayed while we investigated the root cause. Web requests (HTTPS) from Redox were unaffected. We have mitigated the issue and VPN tunnels are back online. If you have any additional questions, please notify us at [email protected].

5. resolved Feb 23, 2026, 09:14 PM UTC

   This incident has been resolved.

6. postmortem Mar 02, 2026, 03:28 PM UTC

   ### **Summary** On **February 23, 2026**, between **1:05 AM to 2:05 AM CST** and **1:53 PM to 3:03 PM CST**, a subset of customers experienced elevated latency and connectivity issues across a number of our MLLP VPN Gateways. Service was fully restored once primary gateway instances were cycled and traffic successfully failed over to secondary nodes. ### **What Happened** The outage was triggered by a memory exhaustion event on specific VPN Gateway instances. There was a memory leak in a third party library led to a steady rise in memory consumption, eventually causing the primary instances to become unresponsive. ### **What We Are Doing** To prevent a recurrence and improve our response time, our engineering team is implementing the following measures: * **Stability:** We are deploying updates that mitigate the issue we saw with the third party memory leak to prevent resource exhaustion and non-responsive primary gateway instances. We submitted a ticket with the third party library and they have resolved the issue in the library. * **Enhanced Monitoring:** We are deploying updates to have more granular alerts that monitor MLLP gateway traffic health and reduce time to detection for unhealthy gateways. * **Service Resilience:** We are reviewing our automated health monitoring and failover protocols to ensure the system responds effectively to instance health changes.

Read the full incident report →

Started Feb 23, 2026, 04:16 PM UTC· resolved Feb 23, 2026, 05:31 PM UTC

1. investigating Feb 23, 2026, 04:16 PM UTC

   At approximately 10:15 ET, Redox became aware of an issue in which logs are not properly displaying on the dashboard. You may notice up to an hour of latency in viewing logs. What this means: While logs are still processing properly, the dashboard is not displaying them as expected. Search within the dashboard will not work as expected until a fix has been implemented. If you have questions about specific logs or your connection and whether it has been affected, please contact us at [email protected]

2. resolved Feb 23, 2026, 05:31 PM UTC

   A fix has been implemented for the log visibility issues and the latency is fully caught up. Logs should now be visible across the entire Redox platform. If you have any addition questions or are still experiencing issues, please reach out to [email protected]

Read the full incident report →

Started Feb 23, 2026, 07:09 AM UTC· resolved Feb 23, 2026, 07:51 AM UTC

1. investigating Feb 23, 2026, 07:09 AM UTC

   At approximately 12:30AMCT, Redox became aware of an issue with our message processing for a subset of customers. What we will do: After a fix has been implemented, Redox will go through affected messages and determine what action can be taken to rectify the situation If you have any additional questions please reach out to us at [email protected]

2. monitoring Feb 23, 2026, 07:30 AM UTC

   A fix has been implemented for the traffic processing issue. Messages are processing properly again. If you have any questions, please contact us at [email protected]

3. resolved Feb 23, 2026, 07:51 AM UTC

   Messages are now processing normally. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]

4. postmortem Mar 02, 2026, 03:29 PM UTC

   ### **Summary** On **February 23, 2026**, between **1:05 AM to 2:05 AM CST** and **1:53 PM to 3:03 PM CST**, a subset of customers experienced elevated latency and connectivity issues across a number of our MLLP VPN Gateways. Service was fully restored once primary gateway instances were cycled and traffic successfully failed over to secondary nodes. ### **What Happened** The outage was triggered by a memory exhaustion event on specific VPN Gateway instances. There was a memory leak in a third party library led to a steady rise in memory consumption, eventually causing the primary instances to become unresponsive. ### **What We Are Doing** To prevent a recurrence and improve our response time, our engineering team is implementing the following measures: * **Stability:** We are deploying updates that mitigate the issue we saw with the third party memory leak to prevent resource exhaustion and non-responsive primary gateway instances. We submitted a ticket with the third party library and they have resolved the issue in the library. * **Enhanced Monitoring:** We are deploying updates to have more granular alerts that monitor MLLP gateway traffic health and reduce time to detection for unhealthy gateways. * **Service Resilience:** We are reviewing our automated health monitoring and failover protocols to ensure the system responds effectively to instance health changes.

Read the full incident report →

Started Feb 10, 2026, 12:50 PM UTC· resolved Feb 10, 2026, 03:13 PM UTC

1. monitoring Feb 10, 2026, 12:50 PM UTC

   At approximately 6:25am ET, Redox became aware of an issue with polling workflows for third-party APIs. What this means: Any messages that would have been generated by these polling workflows would see latency in being delivered to customers by up to an hour. What we're doing: We have implemented a fix and are monitoring the results. You may continue to experience delays as queues catch up. Please contact us at [email protected] if you have any questions.

2. resolved Feb 10, 2026, 03:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 06, 2026, 07:19 PM UTC· resolved Feb 06, 2026, 07:40 PM UTC

1. investigating Feb 06, 2026, 07:19 PM UTC

   We are currently experiencing an issue with increased error rates for location queries to Carequality What this means: If you query Carequality you may see an increased error rate. What we're doing: We are actively investigating the issue and will report updates as we have them. If you have questions Please contact us at [email protected]

2. monitoring Feb 06, 2026, 07:32 PM UTC

   A fix has been implemented to resolve the errors we were getting. We're actively monitoring the situation to ensure our errors return to nominal rates. If you have any additional questions, please contact [email protected]

3. resolved Feb 06, 2026, 07:40 PM UTC

   The incident has been resolved and the Redox Engine has resumed normal operations. If you have any additional questions please reach out to [email protected]

Read the full incident report →

Started Jan 14, 2026, 03:38 PM UTC· resolved Jan 14, 2026, 03:50 PM UTC

1. monitoring Jan 14, 2026, 03:38 PM UTC

   At approximately 8:30am CT, Redox became aware of an issue in which some logs were delayed. What this means: you may have experienced a delay before receiving traffic from Redox. However, please be aware that the actual processing of logs was unaffected; message flow was still occurring, albeit at a slower pace. A fix has been implemented for the delay issues and Redox is actively monitoring our logs. You may continue to experience delays as queues catch up to the present. If you are concerned that your logs have been impacted or have any additional questions, please reach out to [email protected]

2. resolved Jan 14, 2026, 03:50 PM UTC

   The incident has been resolved and logs delays are declining; the delay is expected to be gone by end of day. If you have any additional questions please reach out to [email protected]

Read the full incident report →

Started Dec 01, 2025, 04:20 PM UTC· resolved Dec 01, 2025, 08:33 PM UTC

1. investigating Dec 01, 2025, 04:20 PM UTC

   At approximately 9:39 AM CST, Redox became aware of an issue in which logs are not properly displaying on the dashboard. In addition, a small subset of customer alerting messages are not processing in a timely manner What this means: While logs are still processing properly, the dashboard is not displaying them as expected. Search within the dashboard will not work as expected until a fix has been implemented. If you have questions about specific logs or your connection and whether it has been affected, please contact us at [email protected]

2. investigating Dec 01, 2025, 04:20 PM UTC

   We are continuing to investigate this issue.

3. investigating Dec 01, 2025, 07:18 PM UTC

   We are still actively looking into this issue. We have made progress towards investigating the root cause and are still considering viable solutions. We will continue to update as we have more information.

4. monitoring Dec 01, 2025, 07:33 PM UTC

   A fix has been implemented for the log visibility issues. We are actively monitoring as the delayed logs continue to catch up. We expect all logs to be visible in around 30 minutes. We will update here once all logs are back to visible. If you have any questions please reach out to [email protected]

5. resolved Dec 01, 2025, 08:33 PM UTC

   Logs are now be visible across the entire Redox platform. If you have any addition questions or are still experiencing issues, please reach out to [email protected]

Read the full incident report →

Started Oct 29, 2025, 04:06 PM UTC· resolved Oct 29, 2025, 06:41 PM UTC

1. investigating Oct 29, 2025, 04:07 PM UTC

   At approximately 10:37 AM Central Time, Redox became aware of an issue with our message processing. The exact number of messages impacted is currently unknown What this means: Messages are presently not processing as expected. This appears to be impacting both synchronous and asynchronous messages. Please expect delays in message receipt until we have identified the issue and implemented a fix. What we will do: After a fix has been implemented, Redox will go through affected messages and determine what action can be taken to rectify the situation If you have any additional questions please reach out to us at [email protected]

2. identified Oct 29, 2025, 04:16 PM UTC

   We have identified the cause for the message processing issues. This is related to 3rd party vendors experiencing interruptions. They seem to be recovering and our systems are getting working through getting caught up. We are actively working with these vendors to help monitor and resolve the issue. We will update the status of this incident when said fix has been deployed.

3. monitoring Oct 29, 2025, 04:50 PM UTC

   Our third party vendor has stabilized. Message processing is now performant and our queues are caught up. We will continue monitoring with the help from our vendors. If you have any questions, please contact us at [email protected]

4. resolved Oct 29, 2025, 06:41 PM UTC

   Messages are now processing normally. If you have any additional questions or are not seeing expected logs, please reach out to [email protected]

Read the full incident report →

Started Oct 27, 2025, 01:28 PM UTC· resolved Oct 27, 2025, 03:16 PM UTC

1. identified Oct 27, 2025, 01:28 PM UTC

   Our service sending alert emails is currently paused What this means: Alert emails are not being sent as expected. These include alerts sent out automatically to the alert contact, via lemonade stand and via alert rules. What we're doing: We are working on releasing the queue which will result in multiple alert emails going out at once and then returning to real time alerts. If you have questions, please contact us at [email protected]"

2. identified Oct 27, 2025, 01:28 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Oct 27, 2025, 02:24 PM UTC

   A fix has been deployed and the alert emails are caught up. We are monitoring to ensure continued expected email traffic. If you have questions about the alerts you received feel free to reach out to support at [email protected]

4. resolved Oct 27, 2025, 03:16 PM UTC

   The incident has been resolved and the Redox Engine has resumed normal operations. If you have any additional questions please reach out to [email protected]

Read the full incident report →

Started Sep 30, 2025, 04:54 PM UTC· resolved Sep 30, 2025, 06:23 PM UTC

1. identified Sep 30, 2025, 04:54 PM UTC

   We are currently aware of an issue in which the the last one hour of logs are not properly displaying on the dashboard. Some processing for synchronous traffic may also be delayed. What this means: While asynchronous logs are still processing properly, the dashboard is not displaying them as expected. Search within the dashboard will not work as expected until a fix has been implemented. Additionally, synchronous logs may take longer to process than normal. If you have questions about specific logs or your connection and whether it has been affected, please contact us at [email protected]

2. monitoring Sep 30, 2025, 06:01 PM UTC

   A fix has been implemented for the log visibility issues and most logs should now be visible. We are actively monitoring the situation and ensuring that all affected customers' logs are visible again. The previously mentioned processing delays have been resolved. If you have questions or are still experiencing issues, please reach out to [email protected]

3. resolved Sep 30, 2025, 06:23 PM UTC

   Logs should now be visible across the entire Redox platform. If you have any addition questions or are still experiencing issues, please reach out to [email protected]

4. postmortem Oct 23, 2025, 12:42 AM UTC

   ## **Summary** At approximately 10:39AM CT, September 29 2025, we observed system-wide degradation creating latency for transaction processing and observability. Latency was resolved for most customers by about 13:30PM CT, Sep 29, 2025. At 4:43AM CT, September 30 2025, we observed a similar system degradation. Latency was resolved for customers by 13:23PM CT, September 30 2025. ## **What Happened** * A confluence of several issues generated a number of errors in our system. These factors included a change in the way we manage database partition rotations, an upgrade of a core library, and the interaction of some services with kubernetes. * These errors caused some services to crash, and although our infrastructure is fault-tolerant and can recover from such crashes, the degree of crashing put pressure on our infrastructure in a way that manifested as latency in transaction processing, and a delay in transaction observability. ## **What we are doing about this:** * We rolled back the upgrade made to the affected third-party library. * We are adding and improving alerting on affected services to provide clearer visibility to the teams that are responsible for those services.

Read the full incident report →

Started Sep 30, 2025, 12:30 AM UTC· resolved Sep 30, 2025, 12:30 AM UTC

1. resolved Sep 30, 2025, 04:33 PM UTC

   Redox was made aware of an incident in which Flat File SFTP messages were succeeding but not processing correctly. A fix has already been implemented and all traffic is now up to date and processed. If you have any additional questions please reach out to [email protected]

Read the full incident report →

Started Sep 10, 2025, 01:33 PM UTC· resolved Sep 10, 2025, 01:41 PM UTC

1. identified Sep 10, 2025, 01:33 PM UTC

   We have identified an issue that is creating increased latency in message processing for some customers. We are currently working on implementing a fix.

2. resolved Sep 10, 2025, 01:41 PM UTC

   We have implemented a fix and latency has returned to normal. This incident has been resolved.

Read the full incident report →

Started Sep 08, 2025, 04:19 PM UTC· resolved Sep 08, 2025, 05:54 PM UTC

1. investigating Sep 08, 2025, 04:19 PM UTC

   Message processing impacted for a subset of customers. We are currently investigating the issue.

2. identified Sep 08, 2025, 04:25 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 08, 2025, 04:59 PM UTC

   A fix has been implemented and we are monitoring the results. Message processing is back to normal, and we are replaying affected messages on all impacted connections.

4. resolved Sep 08, 2025, 05:54 PM UTC

   All affected traffic has been replayed. This issue has been resolved.

Read the full incident report →

Started Aug 13, 2025, 01:02 AM UTC· resolved Aug 13, 2025, 01:21 AM UTC

1. identified Aug 13, 2025, 01:02 AM UTC

   Viewing logs in the dashboard is degraded. The actual processing of logs is unaffected. The issue has been identified and a fix is currently being implemented.

2. monitoring Aug 13, 2025, 01:18 AM UTC

   A fix has been implemented and we are starting to see logs gaining visibility again in the dashboard. We expect for the issue to be completely resolved within the half hour.

3. resolved Aug 13, 2025, 01:21 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 11, 2025, 07:46 PM UTC· resolved Aug 11, 2025, 08:10 PM UTC

1. identified Aug 11, 2025, 07:46 PM UTC

   This has been identified and a fix is being implemented.

2. resolved Aug 11, 2025, 08:10 PM UTC

   This incident has been resolved.

Read the full incident report →