Root experienced a minor incident on October 26, 2023 affecting API and Dashboard, lasting 13h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 26, 2023, 07:33 AM UTC
Incident: Slow API Response Times for South African Multi-tenant Issue: Our South African Multi-tenant service is currently experiencing slow API response times. This may lead to delays in data retrieval and other dependent operations for our users, we are currently investigating the issue
- investigating Oct 26, 2023, 07:33 AM UTC
Incident: Slow API Response Times for South African Multi-tenant Cause: The core issue was identified as a performance degradation in our database after a recent version upgrade. Action: We have added additional resources to maintain an acceptable level of system performance for the day and have planned after-hours maintenance to re-build the affected indexes.
- investigating Oct 26, 2023, 05:06 PM UTC
South African Multi-tenant is stable and we are continuing to monitor the performance
- investigating Oct 26, 2023, 06:21 PM UTC
South African Multi-tenant is stable and we are continuing to monitor the performance
- investigating Oct 26, 2023, 06:59 PM UTC
South African Multi-tenant is stable and we are continuing to monitor the performance
- investigating Oct 26, 2023, 07:00 PM UTC
Maintenance Update Action: We are currently performing maintenance to optimize our system performance. Our team is actively rebuilding the affected indexes. This action addresses the root cause of the performance degradation issue.
- investigating Oct 26, 2023, 07:35 PM UTC
Maintenance Update Action: The affected indexes have been rebuilt. We are thoroughly testing the system to confirm the improved post-maintenance performance.
- resolved Oct 26, 2023, 07:37 PM UTC
The issue has been resolved.