Redox experienced a minor incident on October 28, 2024 affecting Dashboard Tools, lasting 9h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 28, 2024, 09:54 PM UTC
We are aware that dashboard search functionality is extremely limited at the moment. Present workaround is not to use the 'all' category when searching, but we are actively working on a fix.
- identified Oct 28, 2024, 10:18 PM UTC
We have identified the issue and are working on a solution presently
- monitoring Oct 28, 2024, 11:21 PM UTC
We've applied a fix in production and are actively monitoring log searches to ensure that full functionality has been restored
- resolved Oct 29, 2024, 07:31 AM UTC
This incident has been resolved.
- postmortem Nov 12, 2024, 04:00 PM UTC
## Summary * From October 25-28, 2024 some Logs searches initiated by use of the search bar errored. ## What Happened * On October 25 we implemented a code change, which was tested in isolation. * On October 28 we were alerted to search errors and started an incident at 1647CT. * At 1710CT we identified the issue and started working on a fix. * At 1819CT the fix was fully deployed and functionality in Logs search was restored. ## What are we doing about it? * We are considering expanding our testing capabilities to account for earlier detection and mitigation of this type of failure in the future.