LogRocket incident
Performance issues with primary search database
LogRocket experienced a major incident on January 18, 2023 affecting Dashboard and Issues and 1 more component, lasting 1d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 18, 2023, 03:41 PM UTC
We are currently investigating an issue impacting our primary search database. Searching for sessions, loading metric charts, issues and alerting are all impacted. Session data collection has not been impacted.
- identified Jan 18, 2023, 04:24 PM UTC
We have identified the issue with our database vendor and are working with them to remedy the situation.
- identified Jan 18, 2023, 06:00 PM UTC
We are continuing to work with our vendor to resolve the issue with ingesting new data.
- identified Jan 18, 2023, 08:02 PM UTC
We are continuing to work with our vendor to resolve the issue.
- identified Jan 19, 2023, 01:00 AM UTC
We are continuing to work with our vendor to remedy the situation. Session data collection continues to be unaffected, but secondary indexing that powers search and metrics continues to be delayed.
- identified Jan 19, 2023, 05:03 AM UTC
We have restored some indexing to the database but are continuing to fully resolve the situation.
- monitoring Jan 19, 2023, 08:15 AM UTC
We have identified the cause of the issue, applied a fix and have begun processing the backlog of events.
- monitoring Jan 19, 2023, 03:41 PM UTC
We have completed processing of our primary indexing backlog, session filtering and metrics will be mostly up-to-date. Our secondary processing system is currently processing and includes some metrics, issues and alerting.
- resolved Jan 20, 2023, 02:00 PM UTC
We've seen no additional performance degradation after our mitigation work.