Service degradation - Admin Panel

Update timeline

1. investigating Dec 21, 2025, 04:32 PM UTC

   For US organizations, there are currently issues with booking trips. Our engineering team is actively investigating the cause of this problem.

2. investigating Dec 21, 2025, 04:35 PM UTC

   For US organizations, there are currently issues with booking/canceling/editing trips. Affected users would see some occasional errors in the product.

3. investigating Dec 21, 2025, 04:46 PM UTC

   We are currently investigating reports of occasional errors affecting some users in the US Region. These issues began at approximately 8:10 AM PT. Our engineering team is actively diagnosing the root cause to restore full stability. Users may experience temporary latency or intermittent failure messages while navigating the product. We will provide further updates as more information becomes available.

4. investigating Dec 21, 2025, 05:00 PM UTC

   We are currently investigating reports of occasional errors affecting some users in the US Region. These issues began at approximately 8:10 AM PT. Our engineering team is actively diagnosing the root cause to restore full stability. Users may experience temporary latency or intermittent failure messages while navigating the product. We will provide further updates as more information becomes available.

5. identified Dec 21, 2025, 05:15 PM UTC

   We have identified the cause of the intermittent errors and are currently taking corrective action. Our team is restarting specific backend services to restore stable connections to our database. We are monitoring the impact of these restarts closely. While we do not have a firm ETA for a full resolution, we expect to see service stability improve as these processes complete. Thank you for your continued patience.

6. resolved Dec 21, 2025, 05:20 PM UTC

   A fix has been implemented, and we have seen service levels return to normal as of 9:10 AM PT. We have successfully restored backend database connections, and the intermittent errors in the US region have subsided. Our engineering team will continue to monitor the system closely for the next hour to ensure continued stability. If you are still experiencing issues, please try refreshing your browser or contact our support team.

7. postmortem Dec 24, 2025, 05:52 PM UTC

   # Spare Incident Report December 24, 2025 ## Executive Summary Between December 21 and December 23, 2025, the Spare platform experienced brief service disruptions impacting the US region. Users may have experienced intermittent error messages and system slowness during several distinct time windows. The disruption was caused by resource contention within our primary database infrastructure, which led to connection pool exhaustion. This prevented backend services from successfully processing queries. Our engineering team intervened immediately during the initial event to restore service. Subsequent brief recurrences were handled by automated recovery systems while a permanent fix regarding database configuration was explored and some changes to the system were implemented. ## Incident Details The incident began on Sunday, December 21, at 8:00 am PST when internal monitoring systems alerted our engineering team to a spike in database connection errors and increased latency in the US region. ### **Cause** The primary root cause for this incident was the exhaustion of the database connection pool, preventing the platform from establishing necessary connections to the database. Upon deeper investigation, it was determined that specific database triggers were acquiring exclusive locks on key resources. This locking behavior created a bottleneck, leading to query backups and eventual exhaustion of available connections. ### **Mitigation and Resolution** Our team’s response focused on immediate service restoration, data integrity verification, and root cause elimination: 1. **Immediate Restoration:** During the initial incident, the engineering team restarted backend services to reset database connections and clear internal caches, restoring full service. 2. **Data Integrity Check:** While our priority was service restoration, we conducted a thorough review out of an abundance of caution to ensure data integrity. We confirmed that all data remained secure and accurate throughout the incident. 3. **Root Cause Elimination:** During the subsequent brief recurrences on December 22 and 23, the team gathered necessary diagnostic data to identify the problematic database triggers. These triggers have been removed to eliminate the resource locking issue and prevent recurrence. ## Timeline _All times are in Pacific Time \(PST\)._ * **Dec 21, 08:00 am:** Monitoring systems detect elevated error rates and latency in the US region. * **Dec 21, 08:03 am**: Team begins incident investigation. * **Dec 21, 09:10 am:** Service is fully restored following a restart of backend services. * **Dec 22, 05:30 am:** A recurrence of connection pool exhaustion is detected. * **Dec 22, 05:38 am:** System auto-recovers as backend services stabilize; diagnostic data is gathered. * **Dec 22, 10:15 am:** A second minor recurrence is detected. * **Dec 22, 10:21 am:** System auto-recovers and normal service resumes. * **Dec 22, 1:35 pm:** Root cause identified \(database triggers\) and implementation of fix begins. * **Dec 23, 8:16 am**: A third minor recurrence occurs. * **Dec 23, 8:29 am**: System auto-recovers and normal service resumes. * **Dec 23, 9:26 am**: Fix rolled out to all systems in all regions. ## Next Steps To ensure the continued reliability of our platform and prevent similar resource contention issues, we are undertaking the following: * **Database Optimization:** We have removed the problematic triggers and are reviewing all database locking mechanisms to ensure optimal concurrency. * **Enhanced Monitoring:** We are implementing stricter alerts regarding database connection pool usage to detect potential contention before it impacts user experience. * **Cache Resiliency:** We are reviewing our caching logic to ensure it recovers more gracefully during periods of backend instability. We appreciate your understanding as we worked through this incident. We remain vigilantly focused on continuous improvement of our system’s stability and availability.