Resolved after 4 hours 1 minute, affecting [United States 1], on October 14, 2025.
# Spare Incident Report October 14, 2025 ## Executive Summary On October 14, 2025, the Spare platform experienced a significant service disruption in the US region, resulting in application-wide slowdowns, errors, and timeouts. This primarily impacted our customers' ability to view Live Requests or access historical request data. Drivers also experienced intermittent issues and slowdowns in the field. The root cause was a confluence of two events: an automated failover on our US region's primary database coinciding with an internal platform rollout. This combination led to a massive spike in concurrent database connection requests, which overwhelmed a core connection pooling service, causing it to reach 100% capacity and block application services from accessing the database. Our engineering team addressed the issue by immediately rolling back the recent changes and implementing a successful critical mitigation: rerouting key high-traffic services to bypass the overloaded connection pooler and connect directly to the database. This action led to initial service recovery at 2:19 pm PDT. However, due to lingering instability in background job processes, the full internal resolution and confirmation of system stability was finalized at 4:37 pm PDT. ## Incident Details The incident began with monitoring alerts and reports of general application sluggishness and timeouts just before noon PDT. Several customer organizations reported difficulties in managing requests and driver activity. ### **Cause** Our investigation determined that the system instability was triggered by two factors occurring nearly simultaneously: 1. **Primary Database Failover:** The core production database automatically failed over to a new primary node. 2. **Internal Rollout:** A rollout of recent changes on two critical backend services completed around the same time. The combination of the database changing primary nodes and the high volume of connection requests from the newly deployed services placed an unsustainable load on our database connection pooling service. This service, designed to manage database access efficiently, became the bottleneck, reaching its CPU limit and denying connections to the rest of the application. ### **Mitigation and Resolution** Our team’s resolution strategy focused on reducing the connection load on the critical pooling service: 1. **Rollback and Stabilization:** We immediately initiated rollbacks on the recently deployed services to reduce immediate pressure and restarted several core components. 2. **Direct Connection Reroute \(Primary Fix\):** The most effective action was modifying the two high-traffic services to temporarily bypass the connection pooling service and connect directly to the main database. This action immediately freed up the overloaded component, reducing connection queueing and allowing application services to resume normal operation. This led to initial service recovery at 2:19 pm PDT. 3. **Capacity Tuning:** Following the initial recovery, we monitored the platform closely and performed extensive fine-tuning of connection pool sizes for background job services to eliminate residual errors and lock in long-term stability. Full functionality and system stability were confirmed at 4:37 pm PDT, and the incident was officially resolved. ## Timeline _All times are in Pacific Time \(PDT\)._ * **11:59 am:** Initial service alerts were received, coinciding with reports of general slowdown and inability to load Live Requests in the US region. * **12:14 pm:** Investigation confirms a failover on the primary US region database had occurred, concurrent with a recent internal rollout. * **1:34 pm:** Confirmed that the connection pooling service was saturated, operating at 100% CPU, and acting as the primary bottleneck. * **2:00 pm:** Engineers begin implementing the critical mitigation step: redirecting high-traffic services to connect directly to the database. * **2:19 pm:** Initial recovery observed; major customer-facing services return to a stable state. * **2:20 pm - 4:37 pm:** Team actively monitors the environment and performs fine-tuning of service connection limits to eliminate residual instability in background processes. * **4:37 pm:** Full functionality is confirmed to be restored across all US region services. The incident is declared resolved. ## Next Steps As part of our commitment to maintaining a robust platform, we will be conducting a comprehensive post-incident review to prevent recurrence. Key areas of focus include: * **Database Failover Investigation:** While the failover mechanism itself performed as designed, we will investigate the underlying cause of the primary database crash/failover to address the root trigger for this event. * **Connection Management Review:** We will review the default configuration and capacity limits of our database connection pooling service to ensure it can gracefully handle simultaneous stress events like database failovers and service rollouts without saturation. * **Service-Level Resilience:** We will permanently implement changes to core services to better handle high connection volume and explore options for services to manage their own connections, reducing dependency on a single point of failure. * **Resource Allocation:** We will perform a detailed audit of resource limits for high-usage services to prevent resource-constrained restarts during peak load conditions. Thank you for your patience and understanding as we navigated this challenging issue. We are fully committed to leveraging the learnings from this incident to continue strengthening the resilience of the Spare platform.
Affected components
Update timeline
- investigating Oct 14, 2025, 07:09 PM UTC
We are currently investigating a service disruption affecting the Spare platform in the US1 (United States 1) region. Users attempting to access or perform actions within the platform are seeing an authorization error. This issue started approximately seven minutes ago. Our Engineering and Support teams are actively working to identify the cause of the issue and implement a fix as quickly as possible. We will share our next update within 15 minutes or as soon as new details become available. We appreciate your patience and understanding as we work to restore full service.
- investigating Oct 14, 2025, 07:21 PM UTC
We are still actively investigating the ongoing service disruption impacting the Spare platform in the US1 (United States 1) region. Our Engineering team is performing key diagnostic steps, including the restart of core authentication services, as we work to identify and stabilize the root cause. All US1 users remain unable to access or perform actions due to the authorization errors. There are no new developments or workarounds at this time, but we will continue sharing updates every 15 minutes or as soon as new information becomes available. We truly appreciate your continued patience while our team works toward a resolution.
- identified Oct 14, 2025, 07:33 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 region. Our team is focused on deploying a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 07:51 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 (United States 1) region. Our team is focused on finishing the deployment of a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the token creation issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. At the moment, we have seen some improvements when entering the Admin Platform. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 08:09 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 (United States 1) region. Our team is focused on finishing the deployment of a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the token creation issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. At the moment, we have seen some improvements when entering the Admin Platform. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 08:20 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 (United States 1) region. Our team is focused on finishing the deployment of a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the token creation issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. At the moment, we have seen some improvements when entering the Admin Platform. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 08:40 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 (United States 1) region. Our team is focused on finishing the deployment of a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the token creation issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. At the moment, we have seen some improvements when entering the Admin Platform. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 08:50 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 (United States 1) region. Our team is focused on finishing the deployment of a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the token creation issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. At the moment, we have seen some improvements when entering the Admin Platform. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 09:07 PM UTC
We are continuing to work actively on the service disruption impacting the Spare platform in the US1 (United States 1) region. Our team is focused on finishing the deployment of a revised stabilization action related to the authentication database, following an earlier database failover. The goal is to correct the token creation issues that are causing users to see authorization errors. We anticipate that if this fix is successful, users should see immediate improvement in access and functionality. We are working as fast as possible to apply this remedy. At the moment, we have seen some improvements when entering the Admin Platform. We will share our next update within 15 minutes or as soon as we have confirmation on the success of the deployment. We truly appreciate your patience as we work through this stabilization process.
- monitoring Oct 14, 2025, 09:27 PM UTC
A fix has been successfully deployed, and we are seeing significant recovery from the service disruption affecting the Spare platform in the US1 region. Most users should now be able to access the platform. However, we are currently observing application slowness and a possibility of intermittent timeout errors as the system returns to full stability. Our Engineering team is continuing to work diligently to eliminate this lingering performance degradation. If you encounter an error message, please refresh your application or browser. We are actively monitoring the situation to ensure full stability and will post another update within 15 minutes or as soon as performance returns to normal. Thank you for your continued patience.
- monitoring Oct 14, 2025, 09:54 PM UTC
A fix has been successfully deployed, and we are seeing significant recovery from the service disruption affecting the Spare platform in the US1 region. Most users should now be able to access the platform. However, we are currently observing application slowness and a possibility of intermittent timeout errors as the system returns to full stability. Our Engineering team is continuing to work diligently to eliminate this lingering performance degradation. If you encounter an error message, please refresh your application or browser. We are actively monitoring the situation to ensure full stability and will post another update within 15 minutes or as soon as performance returns to normal. Thank you for your continued patience.
- monitoring Oct 14, 2025, 09:56 PM UTC
A fix has been successfully deployed, and we are seeing significant recovery from the service disruption affecting the Spare platform in the US1 region. Most users should now be able to access the platform. However, we are currently observing application slowness and a possibility of intermittent timeout errors as the system returns to full stability. Our Engineering team is continuing to work diligently to eliminate this lingering performance degradation. If you encounter an error message, please refresh your application or browser. We are actively monitoring the situation to ensure full stability and will post another update within 15 minutes or as soon as performance returns to normal. Thank you for your continued patience.
- monitoring Oct 14, 2025, 10:11 PM UTC
A fix has been successfully deployed, and we are seeing significant recovery from the service disruption affecting the Spare platform in the US1 region. Most users should now be able to access the platform. However, we are currently observing application slowness and a possibility of intermittent timeout errors as the system returns to full stability. Our Engineering team is continuing to work diligently to eliminate this lingering performance degradation. If you encounter an error message, please refresh your application or browser. We are actively monitoring the situation to ensure full stability and will post another update within 15 minutes or as soon as performance returns to normal. Thank you for your continued patience.
- monitoring Oct 14, 2025, 10:27 PM UTC
No new updates, our team continues to work on solving the problem.
- monitoring Oct 14, 2025, 10:45 PM UTC
No new updates, our team continues to work on solving the problem.
- monitoring Oct 14, 2025, 10:54 PM UTC
No new updates, our team continues to work on solving the problem.
- resolved Oct 14, 2025, 11:14 PM UTC
The service disruption affecting the Spare platform in the US1 region has been fully resolved. All users should now be able to access the platform and perform all actions without issue. The earlier performance degradation, including application slowness and timeout errors, has now cleared following the deployment and stabilization of our fix. We understand that being unable to access the platform due to authorization errors was highly disruptive, and we sincerely apologize for the impact this service disruption had on your day. We will conduct a full review of this incident to prevent recurrence. Thank you for your patience and understanding throughout this incident.
- postmortem Oct 15, 2025, 04:08 PM UTC
# Spare Incident Report October 14, 2025 ## Executive Summary On October 14, 2025, the Spare platform experienced a significant service disruption in the US region, resulting in application-wide slowdowns, errors, and timeouts. This primarily impacted our customers' ability to view Live Requests or access historical request data. Drivers also experienced intermittent issues and slowdowns in the field. The root cause was a confluence of two events: an automated failover on our US region's primary database coinciding with an internal platform rollout. This combination led to a massive spike in concurrent database connection requests, which overwhelmed a core connection pooling service, causing it to reach 100% capacity and block application services from accessing the database. Our engineering team addressed the issue by immediately rolling back the recent changes and implementing a successful critical mitigation: rerouting key high-traffic services to bypass the overloaded connection pooler and connect directly to the database. This action led to initial service recovery at 2:19 pm PDT. However, due to lingering instability in background job processes, the full internal resolution and confirmation of system stability was finalized at 4:37 pm PDT. ## Incident Details The incident began with monitoring alerts and reports of general application sluggishness and timeouts just before noon PDT. Several customer organizations reported difficulties in managing requests and driver activity. ### **Cause** Our investigation determined that the system instability was triggered by two factors occurring nearly simultaneously: 1. **Primary Database Failover:** The core production database automatically failed over to a new primary node. 2. **Internal Rollout:** A rollout of recent changes on two critical backend services completed around the same time. The combination of the database changing primary nodes and the high volume of connection requests from the newly deployed services placed an unsustainable load on our database connection pooling service. This service, designed to manage database access efficiently, became the bottleneck, reaching its CPU limit and denying connections to the rest of the application. ### **Mitigation and Resolution** Our team’s resolution strategy focused on reducing the connection load on the critical pooling service: 1. **Rollback and Stabilization:** We immediately initiated rollbacks on the recently deployed services to reduce immediate pressure and restarted several core components. 2. **Direct Connection Reroute \(Primary Fix\):** The most effective action was modifying the two high-traffic services to temporarily bypass the connection pooling service and connect directly to the main database. This action immediately freed up the overloaded component, reducing connection queueing and allowing application services to resume normal operation. This led to initial service recovery at 2:19 pm PDT. 3. **Capacity Tuning:** Following the initial recovery, we monitored the platform closely and performed extensive fine-tuning of connection pool sizes for background job services to eliminate residual errors and lock in long-term stability. Full functionality and system stability were confirmed at 4:37 pm PDT, and the incident was officially resolved. ## Timeline _All times are in Pacific Time \(PDT\)._ * **11:59 am:** Initial service alerts were received, coinciding with reports of general slowdown and inability to load Live Requests in the US region. * **12:14 pm:** Investigation confirms a failover on the primary US region database had occurred, concurrent with a recent internal rollout. * **1:34 pm:** Confirmed that the connection pooling service was saturated, operating at 100% CPU, and acting as the primary bottleneck. * **2:00 pm:** Engineers begin implementing the critical mitigation step: redirecting high-traffic services to connect directly to the database. * **2:19 pm:** Initial recovery observed; major customer-facing services return to a stable state. * **2:20 pm - 4:37 pm:** Team actively monitors the environment and performs fine-tuning of service connection limits to eliminate residual instability in background processes. * **4:37 pm:** Full functionality is confirmed to be restored across all US region services. The incident is declared resolved. ## Next Steps As part of our commitment to maintaining a robust platform, we will be conducting a comprehensive post-incident review to prevent recurrence. Key areas of focus include: * **Database Failover Investigation:** While the failover mechanism itself performed as designed, we will investigate the underlying cause of the primary database crash/failover to address the root trigger for this event. * **Connection Management Review:** We will review the default configuration and capacity limits of our database connection pooling service to ensure it can gracefully handle simultaneous stress events like database failovers and service rollouts without saturation. * **Service-Level Resilience:** We will permanently implement changes to core services to better handle high connection volume and explore options for services to manage their own connections, reducing dependency on a single point of failure. * **Resource Allocation:** We will perform a detailed audit of resource limits for high-usage services to prevent resource-constrained restarts during peak load conditions. Thank you for your patience and understanding as we navigated this challenging issue. We are fully committed to leveraging the learnings from this incident to continue strengthening the resilience of the Spare platform.
Your product runs on more than Spare
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services