Redox incident
Dashboard logs are up to two hours behind. Processing should not be impacted.
Redox experienced a minor incident on August 25, 2024 affecting Dashboard Tools, lasting 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 25, 2024, 10:03 AM UTC
We are currently investigating this issue.
- monitoring Aug 25, 2024, 10:19 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 25, 2024, 10:49 AM UTC
This incident has been resolved.
- postmortem Sep 05, 2024, 01:17 PM UTC
# Logs intermittently unavailable to view or search ## Summary From August 25-26, 2024 Logs were intermittently delayed or unavailable to view or search in the Redox dashboard. Message processing was unaffected. ## What Happened & How We Responded * On the morning of August 25, AWS initiated an automated failover of their managed database service due to an underlying storage volume issue which subsequently affected throughput of our logs processing. * On August 25th at 0539CT, we restarted impacted application processes which resolved the immediate issue. * On August 26th at 0758CT, we were alerted that logs were again falling behind in processing time. Working with AWS support, we uncovered that the underlying storage from the failover the previous day was still being optimized, resulting in database write latency. The storage optimization completed at 1500CT, and the service was fully available again at 2229CT. ## What we are doing about this: * We are exploring an underlying storage system change to further increase our infrastructure durability.