Nacelle incident

Navigation Data Service Outage

Resolved after 1 hour, on March 10, 2024.

resolved Mar 11, 2024, 07:29 PM UTC

Duration: 3/10 from ~9:00 am - 10:00 am PT (~1 hour) Issue: Starting at ~9:00 am PT on 3/10, some merchant builds failed due to 500 responses when fetching navigation data from the Storefront API. The navigation data service had encountered scaling issues that resulted in a lack of enough replicas being available for the volume of requests. Solution: By ~10:00 am PT, existing self-healing services had allowed navigation data service to stabilize and handle all incoming traffic. As a long-term solution, the number of minimum replicas enabled for the navigation data service has been doubled, and additional monitoring alerts have been implemented for this service.

Started
Mar 10, 2024, 04:00 PM UTC
Resolved
Mar 10, 2024, 05:00 PM UTC
Duration
1h
Detected by Pingoru
Mar 10, 2024, 04:00 PM UTC

Update timeline

  1. resolved Mar 11, 2024, 07:29 PM UTC

    Duration: 3/10 from ~9:00 am - 10:00 am PT (~1 hour) Issue: Starting at ~9:00 am PT on 3/10, some merchant builds failed due to 500 responses when fetching navigation data from the Storefront API. The navigation data service had encountered scaling issues that resulted in a lack of enough replicas being available for the volume of requests. Solution: By ~10:00 am PT, existing self-healing services had allowed navigation data service to stabilize and handle all incoming traffic. As a long-term solution, the number of minimum replicas enabled for the navigation data service has been doubled, and additional monitoring alerts have been implemented for this service.

Your product runs on more than Nacelle

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →