Redox Outage History

Started Aug 11, 2025, 07:08 PM UTC· resolved Aug 11, 2025, 07:20 PM UTC

1. identified Aug 11, 2025, 07:08 PM UTC

   A delay in FHIR.Scheduling data model from HL7 messages have observed to be delayed. A fix is being implemented.

2. resolved Aug 11, 2025, 07:20 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 10, 2025, 02:09 PM UTC· resolved Jul 10, 2025, 08:28 PM UTC

1. identified Jul 10, 2025, 02:09 PM UTC

   Viewing some payload details for a small percent of logs may be delayed. Message processing is not impacted. The issue has been identified and a fix is being implemented.

2. resolved Jul 10, 2025, 08:28 PM UTC

   We have implemented a fix and log visibility is caught up. This incident has been resolved.

Read the full incident report →

Started Jul 07, 2025, 06:26 PM UTC· resolved Jul 07, 2025, 07:29 PM UTC

1. investigating Jul 07, 2025, 06:26 PM UTC

   We are currently investigating this issue.

2. monitoring Jul 07, 2025, 06:55 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jul 07, 2025, 07:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 29, 2025, 03:35 PM UTC· resolved May 29, 2025, 05:34 PM UTC

1. investigating May 29, 2025, 03:35 PM UTC

   We are seeing a number of increased errors with our API and delayed messages of up to 30 minutes. We are currently investigating the issue.

2. identified May 29, 2025, 03:37 PM UTC

   We believe we have identified the root cause of the issue and are deploying a fix.

3. monitoring May 29, 2025, 04:14 PM UTC

   We have implemented a fix and seen that error rates have returned to nominal levels. The message latency is starting to decline. We are continuing to monitor until latency fully resolves.

4. monitoring May 29, 2025, 04:52 PM UTC

   For most feeds we have seen resolved latency. We are continuing to monitor until we see latency fully resolved.

5. resolved May 29, 2025, 05:34 PM UTC

   Latency has now resolved and traffic is flowing as expected.

6. postmortem Jun 11, 2025, 11:19 PM UTC

   ## Summary On May 29, 2025, between 1000CT and 1115CT, some customers experienced elevated error rates and/or delayed processing. The issue impacted some customer traffic during that time, and the duration was <1 1/2 hours. ## What Happened * This issue was caused by a data consistency error with Redox base configs related to change which led to elevated errors and/or delayed processing for customers using a specific version and subset of Redox base configs. * During the incident customers may have experienced an increase in 5XX errors or increased latency, depending on whether traffic was synchronous or asynchronous. * Our team was alerted by monitoring at 1007CT and immediately started investigating. Mitigation efforts included immediately preparing to rollback changes, scaling to increase capacity, and active monitoring of system health to ensure message processing continued. By 1102CT errors were decreasing and latency was starting to return to normal levels Full service was restored by 1115CT. ## What we are doing about this: * Adding automated detection to all environments that would prevent the data consistency errors that caused the incident along with a broader category of data consistency errors. * Improving our rollback capabilities for faster mitigation. * Auditing and improving our standard operating procedure for the process related to working with this type of data.

Read the full incident report →

Started May 28, 2025, 05:32 PM UTC· resolved May 28, 2025, 07:51 PM UTC

1. investigating May 28, 2025, 05:32 PM UTC

   A secondary system we use for our Customer Configured Alerts is currently experiencing an outage. This may result in missed alerts or some extra alerts. Being sent out. This only affects the Customer Configured Alerts not all our other alerting systems. We are monitoring the impact of this secondary system outage.

2. monitoring May 28, 2025, 06:26 PM UTC

   The secondary system's incident is resolved. All Customer Configured Alerts are back up and running. We are investigating possible impact for what alerts may not have been correctly emailed during the outage. You can look at the Monitoring page on your dashboard to see an accurate current status of all customer configured alerts.

3. resolved May 28, 2025, 07:51 PM UTC

   Alerting continues to function as expected. We are reaching out to customers who may have missed emails who are still in an alerted state individually.

Read the full incident report →

Started May 14, 2025, 10:50 PM UTC· resolved May 15, 2025, 02:23 AM UTC

1. identified May 14, 2025, 10:50 PM UTC

   We are aware of an issue with out alerting system that has delayed email alerting due to connection issues by up to two hours. Our development team is implementing a fix, and Redox is actively monitoring the situation.

2. monitoring May 14, 2025, 11:14 PM UTC

   We've implemented a fix and are presently monitoring for further impact

3. resolved May 15, 2025, 02:23 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 12, 2025, 09:14 PM UTC· resolved May 12, 2025, 11:32 PM UTC

1. investigating May 12, 2025, 09:14 PM UTC

   We're currently observing a delay impacting the visibility of the transmission information in some logs. There are NO delays in actual message traffic or delivery — all data is transmitting as expected. Our team is actively working to reduce the latency and will keep you updated on any significant changes. Thank you for your understanding and patience.

2. monitoring May 12, 2025, 10:09 PM UTC

   We have identified the issue and have implemented a change. Log visibility is no longer delayed. We are continuing to monitor the issue as we work on deploying a long term fix.

3. resolved May 12, 2025, 11:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 09, 2025, 05:02 PM UTC· resolved May 09, 2025, 08:18 PM UTC

1. investigating May 09, 2025, 05:02 PM UTC

   We're currently observing a delay affecting the archive importing system, which impacts the visibility of message logs and transmission contents. To clarify: there are no delays in actual message traffic or delivery — all data is transmitting as expected. However, some customers may experience up to a 5-minute delay in viewing these logs within the system. Our team is actively working to reduce the latency and will keep you updated on any significant changes. Thank you for your understanding and patience.

2. investigating May 09, 2025, 05:39 PM UTC

   We are continuing to investigate this issue.

3. investigating May 09, 2025, 05:39 PM UTC

   We are continuing to investigate this issue.

4. monitoring May 09, 2025, 05:40 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 09, 2025, 08:18 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 08, 2025, 12:46 PM UTC· resolved Apr 08, 2025, 01:27 PM UTC

1. identified Apr 08, 2025, 12:46 PM UTC

   Editing filters is currently unavailable in the dashboard. New filters can still be created, and already created filters are still working as intended. Already created filters cannot be edited. The issue has been identified and a fix is being implemented.

2. resolved Apr 08, 2025, 01:27 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 13, 2025, 05:04 PM UTC· resolved Feb 13, 2025, 05:38 PM UTC

1. identified Feb 13, 2025, 05:04 PM UTC

   Two email notification services that Redox uses are down. These services affect emails for notifying customers of expiring certificates and for message traffic or VPN related issues. We have identified the issue and are implementing a fix. When the issue is resolved, customers will receive all backlogged emails that would have been sent while the services were down.

2. monitoring Feb 13, 2025, 05:15 PM UTC

   A fix has been implemented and customers will now start receiving the queued emails. We are continuing to monitor the results.

3. resolved Feb 13, 2025, 05:38 PM UTC

   This incident has been resolved.

4. postmortem Feb 20, 2025, 09:29 PM UTC

   ## Summary On February 11th at 11:30 AM CT, a small number\(<20\) email notifications to users were delayed. These notifications were not sent until February 13th at 12:35 PM CT. ## What Happened * On February 11th, a misconfigured API key in a new notification service prevented email notifications from being sent from a subset of systems. * As a result, customers experienced delays in receiving: * Notifications about traffic errors and resolutions. * Notifications about upcoming certificate expirations \(30 days out\). * The API key was updated and normal processing resumed on February 13th at 12:35 PM CT. ## What we are doing about this: * Additional monitoring and alerting were added to prevent similar potential failures in the future.

Read the full incident report →

Started Feb 06, 2025, 04:48 PM UTC· resolved Feb 06, 2025, 05:36 PM UTC

1. identified Feb 06, 2025, 04:48 PM UTC

   Viewing and editing filters is unavailable in the dashboard. Message processing with filters is not impacted. We have identified the issue and a fix is being implemented.

2. resolved Feb 06, 2025, 05:36 PM UTC

   This incident has been resolved.

3. postmortem Feb 12, 2025, 02:18 PM UTC

   # Summary * On February 6, 2025 at 1025CT we became aware of an issue that affected the ability to view or modify some filters in the Redox dashboard for a subset of customers; at 1136CT we deployed a fix that enabled users to once again interact with filters in the dashboard. * That subset included only customers who had filters that were previously linked to a deleted subscription and there was no impact to filter logic or message processing during this time. ## What Happened * On February 5 we implemented a code change to support an upcoming enhancement to the customer filters experience. * After implementation, we uncovered that, in cases where a subscription had a filter and was later deleted, the user would receive an error; this prevented filters from loading for other subscriptions in that same environment, which meant users could not interact with them for a period of time. * Approximately an hour after we learned of the issue, we deployed a fix and users were again able to interact with filters in the dashboard as expected. ## What we are doing about this: * We are adding additional alerting that would support proactively identifying this type of issue in the future. * We are expanding our understanding of edge cases and incorporating additional testing to support those scenarios.

Read the full incident report →

Started Feb 03, 2025, 11:32 PM UTC· resolved Feb 04, 2025, 12:52 PM UTC

1. investigating Feb 03, 2025, 11:32 PM UTC

   We are aware that some customers may be experiencing increased latency in transmission processing. We are actively looking into this issue, and will update as we know more information

2. investigating Feb 04, 2025, 12:43 AM UTC

   We are aware that some of the affected customers are also receiving NACKs on their transmissions. We are continuing to investigate the root cause of this issue.

3. identified Feb 04, 2025, 05:06 AM UTC

   We have identified the root cause of the issue causing the message latency and are in the process of finding a fix. If you have any questions, please reach out to [email protected]

4. monitoring Feb 04, 2025, 06:10 AM UTC

   We have implemented a fix that is mitigating the effects of the latency we've been seeing. We are actively monitoring to ensure that this fix holds

5. monitoring Feb 04, 2025, 07:18 AM UTC

   We are continuing to monitor the fix that we've implemented. Almost all queues (fewer than a dozen remaining) have drained, and the rest are set to drain by midday today (February 4th). If you have any questions, feel free to reach out to [email protected]

6. resolved Feb 04, 2025, 12:52 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 29, 2025, 03:27 PM UTC· resolved Jan 29, 2025, 04:24 PM UTC

1. investigating Jan 29, 2025, 03:27 PM UTC

   We are investigating an issue that may delay asynchronous traffic log processing for a subset of users.

2. monitoring Jan 29, 2025, 03:37 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 29, 2025, 04:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 24, 2025, 09:14 AM UTC· resolved Jan 24, 2025, 10:06 AM UTC

1. investigating Jan 24, 2025, 09:14 AM UTC

   We are currently seeing slow traffic and increased latency for message processing. We are investigating this issue.

2. monitoring Jan 24, 2025, 09:40 AM UTC

   Traffic and latency seem to be returning to normal, but we are continuing to monitor for further developments

3. resolved Jan 24, 2025, 10:06 AM UTC

   The latency and slowness have gone back to normal levels

4. postmortem Feb 14, 2025, 07:28 PM UTC

   ## Summary Starting January 24, 2025 at 2:40CT we became aware of message processing latency for some of our customers. This latency occurred intermittently through Feb 3 when some message processing stopped, resulting in rejected messages for a subset of customers. A subset of customers with subscriptions in the affected database until the root causes were determined to be from a\(n\) * inefficient query that monitors message processing * lack of monitoring visibility into a set of waiting messages that were in an errored state On February 4, 2025 at 1:15 CT, changes were deployed to fix both root causes were applied with most customers being mitigated by February 4, 2025 17:14 CT. All impacted customer were fully operational by February 5, 2025 at 12:33 CT. ## What Happened * On January 23, atypical messages became stuck in a processing waiting state. Combined with a lack of visibility into errors for that waiting state and an inefficient query for monitoring message processing, one database ran out of available space. * Customers with subscriptions on that database experienced increasing latency intermittently from Jan 24 thru Feb 4. * To mitigate this incident, we removed the problematic messages to unblock customers subscriptions on that one database. Additionally, we made optimizations to the database query that monitors message processing and added metrics to capture and alert the errors from messages waiting to be processed. ## What we are doing about this: * We have created an alert that captures when messages are erroring in this waiting state. * We have corrected the edge case discovered allowing the large message payload. * We have improved performance on a query monitoring message process. * We are improving the process of moving waiting messages into processing to handle atypical messages better.

Read the full incident report →

Started Jan 23, 2025, 05:48 PM UTC· resolved Jan 23, 2025, 09:13 PM UTC

1. identified Jan 23, 2025, 05:48 PM UTC

   We are aware of an issue affecting customer message filters. If you suspect your messages are being unexpectedly filtered, or if your messages should be filtered but are not, please reach out to Production Support here: https://redoxengine.atlassian.net/servicedesk/customer/portal/12

2. monitoring Jan 23, 2025, 06:43 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 23, 2025, 09:13 PM UTC

   This incident has been resolved.

4. postmortem Jan 28, 2025, 03:18 PM UTC

   ## Summary * On January 22, 2025 from ~0900CT to 1650CT some filters that had been previously deleted were reactivated, which may have caused some filter logic to execute in unexpected ways. Message processing was otherwise unaffected. * Secondarily, on January 23, 2025 we identified that for a subset of customers, messages that should have been filtered via Redox logic, were not. No traffic was sent to you that shouldn’t have been sent, but your customer filter configuration may have affected what messages you received or filtered out during this time. This did not affect who receives your message. * All impacted customers were notified directly and offered replay assistance as well as support for assessing further impact. ## What Happened * On January 22 an initial storage system script was executed. This script inadvertently reactivated some previously deleted filters, causing these previously deleted filters to run against subscription traffic during the time period. * On January 22 we were notified of unexpected filters running against traffic. After detection of the inadvertent reactivation, a second script was executed to re-delete the applicable filters. At this point unexpected filtering was mitigated. * On January 23 we were notified that the secondary script, while mitigating the original issue, had caused a secondary issue. A subset of filters which should have been filtering traffic were no longer doing so. A tertiary script was executed to mitigate the secondary issue. * This sequence of events created the following possible impacts, for a subset of customers * On January 22 some traffic which should not have been filtered was. * On January 23 some traffic which should have been filtered instead was not. * On both January 22 and January 23 impacted customers were contacted and assistance was offered to remediate. * Replays were performed, if requested, for the January 22 filtering. ## What we are doing about this: * We are implementing improved alerting to proactively notify us of misaligned expectations in our underlying storage system. * We are considering improvements to our internal tooling for interacting with resources in our underlying storage system.

Read the full incident report →

Started Jan 22, 2025, 10:30 PM UTC· resolved Jan 23, 2025, 12:56 AM UTC

1. investigating Jan 22, 2025, 10:30 PM UTC

   We are aware of an issue affecting customer message filters. If you suspect your messages are being unexpectedly filtered, please reach out to Production Support here: https://redoxengine.atlassian.net/servicedesk/customer/portal/12

2. monitoring Jan 22, 2025, 11:25 PM UTC

   A fix has been implemented and we are working on replaying affected messages

3. resolved Jan 23, 2025, 12:56 AM UTC

   Emails will be sent to affected customers to determine replay eligibility tomorrow morning. Please keep an eye on your alert inbox for updates

4. postmortem Jan 28, 2025, 03:17 PM UTC

   ## Summary * On January 22, 2025 from ~0900CT to 1650CT some filters that had been previously deleted were reactivated, which may have caused some filter logic to execute in unexpected ways. Message processing was otherwise unaffected. * Secondarily, on January 23, 2025 we identified that for a subset of customers, messages that should have been filtered via Redox logic, were not. No traffic was sent to you that shouldn’t have been sent, but your customer filter configuration may have affected what messages you received or filtered out during this time. This did not affect who receives your message. * All impacted customers were notified directly and offered replay assistance as well as support for assessing further impact. ## What Happened * On January 22 an initial storage system script was executed. This script inadvertently reactivated some previously deleted filters, causing these previously deleted filters to run against subscription traffic during the time period. * On January 22 we were notified of unexpected filters running against traffic. After detection of the inadvertent reactivation, a second script was executed to re-delete the applicable filters. At this point unexpected filtering was mitigated. * On January 23 we were notified that the secondary script, while mitigating the original issue, had caused a secondary issue. A subset of filters which should have been filtering traffic were no longer doing so. A tertiary script was executed to mitigate the secondary issue. * This sequence of events created the following possible impacts, for a subset of customers * On January 22 some traffic which should not have been filtered was. * On January 23 some traffic which should have been filtered instead was not. * On both January 22 and January 23 impacted customers were contacted and assistance was offered to remediate. * Replays were performed, if requested, for the January 22 filtering. ## What we are doing about this: * We are implementing improved alerting to proactively notify us of misaligned expectations in our underlying storage system. * We are considering improvements to our internal tooling for interacting with resources in our underlying storage system.

Read the full incident report →

Started Dec 13, 2024, 01:00 PM UTC· resolved Dec 13, 2024, 01:29 PM UTC

1. investigating Dec 13, 2024, 01:00 PM UTC

   Redox is investigating an issue affecting message processing for a subset of customers. Affected customers may notice a stop to inbound traffic from Redox and an inability to send messages outbound to Redox.

2. identified Dec 13, 2024, 01:07 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Dec 13, 2024, 01:13 PM UTC

   A fix has been implemented and we are monitoring the results. Customers should see that traffic flow has resumed and may notice a larger queue depth until caught up.

4. resolved Dec 13, 2024, 01:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 04, 2024, 10:09 PM UTC· resolved Dec 05, 2024, 01:16 AM UTC

1. identified Dec 04, 2024, 10:09 PM UTC

   The issue has been identified and a fix has been implemented. We are now working with the appropriate AdvancedMD teams to have our credentials reset so polling and other workflows can be resumed.

2. resolved Dec 05, 2024, 01:16 AM UTC

   A large majority of polling workflows have been re-enabled and traffic should be flowing as expected. A small subset of credentials do need to be reset but we have communicated with these customers individually to resolve these on a case by case basis. The incident is now considered resolved.

Read the full incident report →

Started Nov 22, 2024, 05:29 PM UTC· resolved Nov 23, 2024, 12:17 AM UTC

1. investigating Nov 22, 2024, 05:29 PM UTC

   This appears to be a broader Carequality issue at this time and we have reached out to Carequality support to help identify and troubleshoot these request errors.

2. investigating Nov 22, 2024, 10:59 PM UTC

   We understand the impacts of this issue and our teams are continuing to collaborate with Carequality and internal partners. Your understanding and patience is appreciated while we continue to work toward a resolution.

3. monitoring Nov 22, 2024, 11:54 PM UTC

   A fix has been implemented. We have updated our end to meet the updated Carequality standard and messages are going through as expected. We will continue to monitor on our end for any additional unexpected changes.

4. resolved Nov 23, 2024, 12:17 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 21, 2024, 10:27 PM UTC· resolved Nov 21, 2024, 10:46 PM UTC

1. monitoring Nov 21, 2024, 10:27 PM UTC

   Connections where we poll external APIs for messages will be delayed by up to 30 minutes. The issue has been Identified and a fix has been released. We are not monitoring.

2. resolved Nov 21, 2024, 10:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 20, 2024, 03:48 PM UTC· resolved Nov 20, 2024, 04:01 PM UTC

1. investigating Nov 20, 2024, 03:48 PM UTC

   Redox is investigating an issue affecting the ability to view Logs in the Dashboard. Log processing is NOT affected. Messages are processing as expected. The visibility of logs in the dashboard is unavailable.

2. identified Nov 20, 2024, 03:56 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Nov 20, 2024, 04:01 PM UTC

   This incident has been resolved. Log visibility is available.

4. postmortem Nov 26, 2024, 03:14 PM UTC

   ## Summary * On November 20, 2024 from 0915CT to 1000CT Logs details were not visible in the dashboard due to logs processing delays. Message processing was not affected. ## What Happened * We made a code change that changed the underlying infrastructure for Logs. * We had planned for this type of failure and as a result were quickly able to fail forward. ## What are we doing about it? * We will expand our internal alerting and monitoring to include an endpoint that would have enabled us to identify this type of issue more quickly.

Read the full incident report →

Started Oct 28, 2024, 09:54 PM UTC· resolved Oct 29, 2024, 07:31 AM UTC

1. investigating Oct 28, 2024, 09:54 PM UTC

   We are aware that dashboard search functionality is extremely limited at the moment. Present workaround is not to use the 'all' category when searching, but we are actively working on a fix.

2. identified Oct 28, 2024, 10:18 PM UTC

   We have identified the issue and are working on a solution presently

3. monitoring Oct 28, 2024, 11:21 PM UTC

   We've applied a fix in production and are actively monitoring log searches to ensure that full functionality has been restored

4. resolved Oct 29, 2024, 07:31 AM UTC

   This incident has been resolved.

5. postmortem Nov 12, 2024, 04:00 PM UTC

   ## Summary * From October 25-28, 2024 some Logs searches initiated by use of the search bar errored. ## What Happened * On October 25 we implemented a code change, which was tested in isolation. * On October 28 we were alerted to search errors and started an incident at 1647CT. * At 1710CT we identified the issue and started working on a fix. * At 1819CT the fix was fully deployed and functionality in Logs search was restored. ## What are we doing about it? * We are considering expanding our testing capabilities to account for earlier detection and mitigation of this type of failure in the future.

Read the full incident report →