Redox experienced a major incident on November 20, 2024 affecting Dashboard Tools, lasting 13m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 20, 2024, 03:48 PM UTC
Redox is investigating an issue affecting the ability to view Logs in the Dashboard. Log processing is NOT affected. Messages are processing as expected. The visibility of logs in the dashboard is unavailable.
- identified Nov 20, 2024, 03:56 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Nov 20, 2024, 04:01 PM UTC
This incident has been resolved. Log visibility is available.
- postmortem Nov 26, 2024, 03:14 PM UTC
## Summary * On November 20, 2024 from 0915CT to 1000CT Logs details were not visible in the dashboard due to logs processing delays. Message processing was not affected. ## What Happened * We made a code change that changed the underlying infrastructure for Logs. * We had planned for this type of failure and as a result were quickly able to fail forward. ## What are we doing about it? * We will expand our internal alerting and monitoring to include an endpoint that would have enabled us to identify this type of issue more quickly.