Redox experienced a minor incident on February 13, 2025 affecting Dashboard Tools, lasting 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 13, 2025, 05:04 PM UTC
Two email notification services that Redox uses are down. These services affect emails for notifying customers of expiring certificates and for message traffic or VPN related issues. We have identified the issue and are implementing a fix. When the issue is resolved, customers will receive all backlogged emails that would have been sent while the services were down.
- monitoring Feb 13, 2025, 05:15 PM UTC
A fix has been implemented and customers will now start receiving the queued emails. We are continuing to monitor the results.
- resolved Feb 13, 2025, 05:38 PM UTC
This incident has been resolved.
- postmortem Feb 20, 2025, 09:29 PM UTC
## Summary On February 11th at 11:30 AM CT, a small number\(<20\) email notifications to users were delayed. These notifications were not sent until February 13th at 12:35 PM CT. ## What Happened * On February 11th, a misconfigured API key in a new notification service prevented email notifications from being sent from a subset of systems. * As a result, customers experienced delays in receiving: * Notifications about traffic errors and resolutions. * Notifications about upcoming certificate expirations \(30 days out\). * The API key was updated and normal processing resumed on February 13th at 12:35 PM CT. ## What we are doing about this: * Additional monitoring and alerting were added to prevent similar potential failures in the future.