Service Disruption Affecting Spare Admin Platform - US1

Update timeline

1. investigating Apr 01, 2026, 05:38 PM UTC

   We are currently investigating an issue affecting access to the US1 (United States 1) Spare Admin platform. Some users may be experiencing difficulties logging in or accessing the platform. Our engineering team has identified the issue and is actively working to restore full access as quickly as possible. We will provide further updates as more information becomes available. Thank you for your patience.

2. investigating Apr 01, 2026, 05:50 PM UTC

   We are currently addressing a service disruption impacting the US1 Spare Admin platform. Helpful resources for reference: • Business Continuity Plan (BCP): https://www.notion.so/189e9a7769a6802185f1e65cc30d5321?pvs=25 • Disaster Recovery – Emergency Response Guide: https://drive.google.com/file/d/1KXtfZy70AGhM48QElJGMvxnwqMAV3Shx/view • Setting up Disaster Recovery Backups: https://help.sparelabs.com/en/articles/10134588-setting-up-disaster-recovery-backups • Using Dispatch Backup Files: https://help.sparelabs.com/en/articles/10263338-using-dispatch-backup-files We are committed to restoring full functionality promptly and will issue a follow-up once the situation is resolved.

3. investigating Apr 01, 2026, 06:00 PM UTC

   There are no updates at the moment - We are currently addressing a service disruption impacting the US1 Spare Admin platform. Helpful resources for reference: • Business Continuity Plan (BCP): https://www.notion.so/189e9a7769a6802185f1e65cc30d5321?pvs=25 • Disaster Recovery – Emergency Response Guide: https://drive.google.com/file/d/1KXtfZy70AGhM48QElJGMvxnwqMAV3Shx/view • Setting up Disaster Recovery Backups: https://help.sparelabs.com/en/articles/10134588-setting-up-disaster-recovery-backups • Using Dispatch Backup Files: https://help.sparelabs.com/en/articles/10263338-using-dispatch-backup-files We are committed to restoring full functionality promptly and will issue a follow-up once the situation is resolved.

4. investigating Apr 01, 2026, 06:15 PM UTC

   There are no updates at the moment - We are currently addressing a service disruption impacting the US1 Spare Admin platform. Helpful resources for reference: • Business Continuity Plan (BCP): https://www.notion.so/189e9a7769a6802185f1e65cc30d5321?pvs=25 • Disaster Recovery – Emergency Response Guide: https://drive.google.com/file/d/1KXtfZy70AGhM48QElJGMvxnwqMAV3Shx/view • Setting up Disaster Recovery Backups: https://help.sparelabs.com/en/articles/10134588-setting-up-disaster-recovery-backups • Using Dispatch Backup Files: https://help.sparelabs.com/en/articles/10263338-using-dispatch-backup-files We are committed to restoring full functionality promptly and will issue a follow-up once the situation is resolved.

5. monitoring Apr 01, 2026, 06:32 PM UTC

   We have applied a fix and are monitoring the Spare Admin platform following the earlier service disruption affecting the US1 environment. At this time, system performance has stabilized and we are not observing ongoing impact. Our engineering team remains engaged and is closely monitoring to ensure full recovery and prevent recurrence. We will continue to provide updates if anything changes.

6. monitoring Apr 01, 2026, 06:45 PM UTC

   We are continuing to monitor the Spare Admin platform following the earlier service disruption affecting the US1 environment. At this time, system performance has stabilized and we are not observing ongoing impact. Our engineering team remains engaged and is closely monitoring to ensure full recovery and prevent recurrence. We will continue to provide updates if anything changes.

7. monitoring Apr 01, 2026, 07:01 PM UTC

   We are continuing to monitor the Spare Admin platform following the earlier service disruption affecting the US1 environment. At this time, system performance has stabilized and we are not observing ongoing impact. Our engineering team remains engaged and is closely monitoring to ensure full recovery and prevent recurrence. We will continue to provide updates if anything changes.

8. resolved Apr 01, 2026, 07:17 PM UTC

   The issue affecting the Spare Admin platform in the US1 environment has been resolved. System performance has returned to normal levels, and users should no longer experience issues accessing the platform. Our team will continue to review this incident internally and conduct a full root cause analysis to help prevent similar disruptions in the future. If you continue to experience any issues, please don’t hesitate to reach out to our support team. Thank you for your patience while we worked to resolve this.

9. postmortem Apr 06, 2026, 04:59 PM UTC

   # Spare Incident Report April 1, 2026 ## Executive Summary On April 1, 2026, the Spare platform experienced a significant service disruption impacting organizations within our US1 data residency region. The incident was characterized by severe system latency, connection timeouts, and elevated error rates, which degraded performance and intermittent unavailability for a subset of users. The disruption originated from database congestion caused by degraded query optimization statistics, resulting in database connection pool saturation. Our engineering team restored service by manually optimizing database statistics and prioritizing critical application traffic by temporarily reducing background workload. Service was fully restored to all affected organizations by 11:35 am PT. ## Incident Details The instability began at approximately 10:30 am PT. Monitoring systems and user reports indicated that core backend services experienced degraded request processing performance resulting in widespread disruption within the US1 region. While other regions remained fully operational, organizations in US1 experienced delays, elevated error rates, and failures to connect to the platform. ### Cause The primary root cause was database connection saturation triggered by inefficient query execution. Detailed investigation identified the following technical factors: * **Stale Query Statistics**: The internal statistics used by the database to plan the most efficient way to retrieve data for Duties became out of date. * **Inefficient Indexing**: Due to these stale statistics, the system began using an inefficient index for a very high-frequency query. * **Resource Exhaustion**: This inefficient processing caused database response times to increase significantly, reaching multiple seconds, quickly exhausting the connection pool and preventing other services from communicating with the database. ### Mitigation and Resolution The engineering team implemented a multi-stage response to stabilize the environment: * **Workload Management**: We throttled recurring background jobs and disabled non-critical health checks to reduce the immediate strain on the database. * **Connection Scaling**: The database connection limit was increased by 50%, which helped stabilize conditions while the root cause was addressed until the underlying query efficiency was addressed. * **Manual Optimization**: The team executed a manual ANALYZE command on the production database. This refreshed the query statistics, allowing the database to resume using the most efficient index. * **Service Restoration**: Once statistics were updated, performance returned to normal levels, and background job throttling was gradually removed. Full restoration of service was confirmed at 11:35 am PT. During this incident, some customers utilized disaster recovery mode built into the Spare platform, which enables service continuity for rider pickups and dropoffs during outages like this one. We are aware of several customers that experienced difficulties with this mode, particularly relating to not being able to login when there were no active duties. We have identified this issue and are prioritizing a fix, with mitigation steps in place in the interim ### Timeline _All times are in Pacific Time \(PT\)._ * 10:30 am: Incident begins; severe latency and 5xx errors detected in US1 region. * 10:35 am: Status page updated; mass communications initiated to affected users. * 10:56 am: Database connection limits increased to attempt mitigation. * 11:00 am: Engineering initiates throttling of non-essential recurring jobs. * 11:22 am: Manual ANALYZE operation initiated, followed by initial signs of recovery as latency begins to drop. * 11:35 am: Full service restored to all US1 organizations. * 11:59 am: All background job throttles removed after confirming sustained stability. ## Next Steps To prevent a recurrence of this specific indexing failure, we are undertaking the following: * **Automated Statistics Refreshes**: We are implementing more aggressive and automated ANALYZE schedules for high-traffic tables \(such as the Duty table\) to ensure query plans remain optimized. * **Query Planner Tuning**: Our team is investigating adjustments to the database planner to ensure the system consistently favors more selective, efficient indexes. * **Resiliency Enhancements**: We are evaluating the implementation of circuit breakers and Redis caching for high-frequency "Duty" lookups to protect the database from total saturation if a query slows down in the future. * **Root Cause Analysis**:We are conducting a detailed root cause validation to ensure all contributing factors are addressed.. We recognize the critical role our platform plays in your daily operations and the significant impact this outage had on your drivers and riders. We sincerely apologize for the disruption and are committed to strengthening our infrastructure to prevent a repeat of this issue, and to consistently achieve the uptime and reliability you expect from Spare.