Unexpected Dashboard Alerting Outage

Update timeline

1. investigating Mar 05, 2026, 08:48 PM UTC

   Our automated monitoring tools detected a partial outage affecting the dashboard at ~2:00 PM Central. Intermittently, alert rules are unable to be loaded within the dashboard and a subset of alerts are not triggering as expected. Our team is currently investigating the cause and working on a solution. Please work with your teams to implement downtime procedures. Traffic processing is not impacted. If you have any additional questions, please notify us at [email protected].

2. identified Mar 05, 2026, 10:25 PM UTC

   We're continuing to investigate the root cause of the issue. It appears to be related to a third-party vendor. We are seeing degraded performance rather than an outage at this time. We are reaching out to this partner to assist with additional troubleshooting. Additionally, we are taking steps to put failovers in place to ensure reliability of our dashboard alerts. A fix is also being put in place to ensure the dashboard still displays as expected.

3. identified Mar 05, 2026, 11:40 PM UTC

   A fix has been put in place for the dashboard so it now reflects current alert statuses as expected. We are continuing to work toward more reliable alert notification services.

4. identified Mar 06, 2026, 02:26 PM UTC

   We have identified the cause for the ongoing notification issue and are actively implementing a fix. We will update the status of this incident when this fix has been fully deployed.

5. identified Mar 06, 2026, 04:48 PM UTC

   The fix was deployed but we are still seeing intermittent errors on our end. We have been communicating with our 3rd party vendor to assist further. We will continue to provide updates as they become available.

6. identified Mar 06, 2026, 04:49 PM UTC

   The fix was deployed but we are still seeing intermittent errors on our end. We have been communicating with our 3rd party vendor to assist further. We will continue to provide updates as they become available.

7. identified Mar 06, 2026, 09:33 PM UTC

   Our third-party vendor is in the process of recovering from an incident on their end. We are doing everything within our power on the Redox side to mitigate from our end. We will continue to provide updates as they become available and will have more information available on Monday. Alerting within the customer dashboard will be inconsistent for now and through the weekend. We recommend reviewing any alerts with scrutiny to ensure they are accurate. Our own Redox internal alerting and remains unimpacted. This includes automated destination level error retry and failure alerts.

8. identified Mar 09, 2026, 02:56 PM UTC

   Our third party vendor has completed their recovery and is monitoring things as are we. Alerts are working as expected and we closing monitoring the situation to ensure this continues. If you have any additional questions, please contact [email protected]

9. monitoring Mar 09, 2026, 02:56 PM UTC

   Our third party vendor has completed their recovery and is monitoring things as are we. Alerts are working as expected and we closing monitoring the situation to ensure this continues. If you have any additional questions, please contact [email protected]

10. resolved Mar 10, 2026, 04:36 PM UTC

    This issue has been resolved as our third party vendor has resolved the issue and our alerts stayed operational through our monitoring process. If you have any additional questions please reach out to [email protected]