Service Disruption: Degraded Performance on Admin Panel and Rider App

Update timeline

1. investigating Aug 08, 2026, 01:03 PM UTC

   We are currently experiencing a service disruption affecting both the admin panel and the rider app. Users accessing these platforms may notice that performance is running significantly slower than usual. Our team is actively working with Engineering to investigate the root cause and restore normal performance levels as quickly as possible. There are currently no workarounds required, but we will share another update within 15 minutes or as soon as new information becomes available. Thank you for your patience and understanding as we work to resolve this.

2. investigating Aug 08, 2026, 01:22 PM UTC

   We are still actively investigating the service disruption affecting the admin panel and the rider app, and our team is continuing to work closely with Engineering to identify a solution. There are no new developments at this time, but we are dedicated to resolving the degraded performance as quickly as possible. We will post another update within 15 minutes or as soon as new information becomes available. We truly appreciate your continued patience and understanding.

3. investigating Aug 08, 2026, 01:37 PM UTC

   We are still actively investigating the service disruption affecting the admin panel and the rider app, and our team is continuing to work closely with Engineering to identify the cause. There are no new developments at this time, but we are fully dedicated to resolving the degraded performance as quickly as possible. We will post another update within the next 15 minutes or as soon as new information becomes available. We truly appreciate your continued patience and understanding.

4. investigating Aug 08, 2026, 01:53 PM UTC

   We are still actively investigating the service disruption affecting the admin panel and the rider app, and our team is continuing to work closely with Engineering to identify a solution. There are no new developments at this time, but we remain fully dedicated to restoring performance as quickly as possible. We will post another update within the next 15 minutes or as soon as new information becomes available. We truly appreciate your continued patience and understanding.

5. investigating Aug 08, 2026, 02:09 PM UTC

   We are still actively investigating the service disruption affecting the admin panel and the rider app, and our team is continuing to work closely with Engineering to identify a solution. There are no new developments at this time, but we remain fully dedicated to restoring performance as quickly as possible. We will post another update within the next 15 minutes or as soon as new information becomes available. We truly appreciate your continued patience and understanding.

6. identified Aug 08, 2026, 02:14 PM UTC

   We have identified the cause of the service disruption affecting the admin panel and the rider app, which was related to a backend capacity issue. Our engineering team is currently pushing forward a system update to allow application services to properly reschedule and restore capacity. Users may continue to experience degraded performance or slow load times while this fix is deployed across the infrastructure. No temporary workarounds or specific actions are required from users at this time. While we do not have an exact estimated resolution time yet, we are monitoring progress closely and will share our next update within 15 minutes. Thank you for your continued patience as we work to restore normal performance levels.

7. monitoring Aug 08, 2026, 02:20 PM UTC

   We have applied a fix, and systems have fully recovered across both the admin panel and the rider app. Both platforms are now operating normally, and response times have returned to optimal levels. Users can seamlessly access and manage their tasks as expected without taking any additional action. We will continue to monitor performance closely to ensure ongoing stability. Thank you so much for your patience and support while we worked to resolve this service disruption!

8. monitoring Aug 08, 2026, 02:36 PM UTC

   We are continuing to monitor performance closely across both the admin panel and the rider app. While systems are currently stable and showing signs of recovery, our engineering team remains actively engaged as we verify full stability across all regions. Users should no longer experience delays, but if any minor issues persist, please try refreshing your browser or restarting the app. We will post another update within the next 15 minutes to confirm full resolution. Thank you for your continued patience and support!

9. resolved Aug 08, 2026, 02:53 PM UTC

   We have successfully resolved the service disruption affecting the admin panel and the rider app, and we can confirm full system stability. Both platforms are operating normally, and performance has returned to optimal levels. You do not need to take any additional actions, such as refreshing your app or logging back in, to continue your usual tasks. Thank you so much for your patience and support throughout this service disruption. We appreciate you bearing with us while our team worked to get everything back on track!

10. postmortem Aug 10, 2026, 05:53 PM UTC

    **SPARE LABS INC. CONFIDENTIAL PROPERTY** # **Spare Incident Report** August 8, 2026 ## **Executive Summary** ‌ On Saturday, August 8, 2026, between 5:10 AM and 7:10 AM Pacific Time, organizations in our US1 data residency region experienced a critical service disruption. The admin panel loaded slowly, and in many cases windows would spin and then return a network error; the rider app was slower and less reliable over the same period. The cause was an incorrect configuration on our side: once a single availability zone had no room available, our US1 API service was prevented from adding capacity anywhere, including in zones that had capacity free. Our engineering team restored service by correcting that configuration, and the correction was rolled out to every region the same day. The platform has been healthy since. ## **Impact** ‌ Measured against customer-facing US1 API traffic: * **Errors.** 7.29% of requests returned a server error across the incident window and at 6:15 AM PT it peaked at 24.34% — roughly one request in four. * **Speed.** Average response time was 4.08s during the incident against 0.53s after recovery. * **Trip Booking impact.** Error rate alone understates the disruption, because slow and intermittently failing screens change how people work. Across the same 5:00–7:10 AM PT window on three Saturdays, successful trip creations were 4,027 on July 25 and 4,255 on August 1, against 1,775 on August 8 — a reduction of more than 50%. * **OTP Impact**: Drivers were still able to continue trip completions and navigation during the incident. Global OTP across US1 region during outage fell to 91.59%, 1.89% lower than a typical Saturday during the same time window. This was a significant degradation of service rather than a total outage. Most requests still succeeded and agencies were able to continue operating, but at materially reduced effectiveness for two hours on a Saturday morning. ## **Cause** ‌ **Underlying cause.** Our API service was configured to keep its instance counts strictly balanced across availability zones. That configuration was wrong for the way our infrastructure operates, and had been in place well before this incident \(since late 2024\) Whenever any single zone had no schedulable room, the service could not place new instances there — and the same rule then forbade placing them in the other zones, including zones with capacity sitting free. A rule intended to make the service resilient to losing a zone instead guaranteed it could not recover from ordinary capacity pressure in one. It was a latent fault waiting for an event that consumed room in a single zone. **What set it off.** A routine rollout of our routing service the previous afternoon increased the memory reserved on US infrastructure nodes, leaving one zone with no schedulable room. That rollout was ordinary and within normal operating parameters; any change that consumed room in a single zone would have produced the same result. The misconfiguration was the failure, not the rollout. **Detection.** Our support team learned of the disruption from customer calls rather than from our own alerting; Our load-balancer error alert was incorrectly defined as a cluster-scoped rule. The alert never fired, and detection fell to customer reports. Reviewing the configuration during the incident, we found two additional alert rules affected in the same way. ## **Mitigation and Resolution** ‌ At 6:55 AM PT the zone-balancing rule was corrected to allow instances to run wherever capacity was available. Existing instances did not pick up the change on their own, so the deployment was advanced manually; sixteen healthy instances came online, server errors dropped sharply, and impact ended at approximately 7:10 AM PT. Both permanent changes shipped the same day. The corrected configuration was applied to all actively maintained release branches and synchronized across every region. The load-balancer alert was converted to a global rule, validated by forcing it to fire and confirming it reached our paging system, then rolled out everywhere. The platform entered the weekend with the fix in place and with working automated detection for a recurrence. ## **Timeline** ‌ _All times are in Pacific Time \(PT\)._ * **5:10 AM** — Customer impact begins. The admin panel slows and rider app errors begin. * **5:45 AM** — Incident declared and investigation begins. * **6:15 AM** — Error rate peaks at 24.34%. New API instances are confirmed unable to be placed on any node. * **6:55 AM** — Zone-balancing rule corrected, allowing instances to be placed in zones with capacity. * **7:05 AM** — Stalled deployment advanced manually; sixteen healthy instances online. * **7:10 AM** — Customer impact ends. ## **Next Steps** ‌ * **Zone-balancing configuration corrected platform-wide** _\(Completed, August 8\)_ — All service deployments now allow placement in any zone with available capacity, so a single constrained zone can no longer prevent a service from adding capacity. * **Load-balancer alerting repaired and audited** _\(Completed, August 8\)_ — The affected alert now uses the correct rule scope and has been verified end to end in every region. We are auditing the remaining alert rules for the same class of misconfiguration. * **Review of deployment placement rules** _\(In progress\)_ — We are reviewing other aspects of our cloud topology configuration for other constraints that could similarly prevent a service from recovering. The infrastructure team will be performing a chaos engineering test in order to uncover additional risks. * **Additional alerting rules** _\(In progress\)_ — We are reviewing additional alerting policy on cluster scheduling rules with the goal of being alerted before agencies start experiencing latency issues or elevated errors. ## **Closing** ‌ We understand what two hours of a slow, unreliable platform costs an operations team on a Saturday morning, and that agencies were calling us before we knew there was a problem — which is not the standard we hold ourselves to. We take full responsibility. This was not an unusual event: it was a configuration on our side that was set up incorrectly and would have caused an outage sooner or later. It has been corrected, along with the alerting gap that let it go undetected. _If you have any questions, please reach out to our support team._