Nacelle experienced a minor incident on March 11, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Mar 11, 2024, 07:29 PM UTC
Duration: 3/10 from ~9:00 am - 10:00 am PT (~1 hour) Issue: Starting at ~9:00 am PT on 3/10, some merchant builds failed due to 500 responses when fetching navigation data from the Storefront API. The navigation data service had encountered scaling issues that resulted in a lack of enough replicas being available for the volume of requests. Solution: By ~10:00 am PT, existing self-healing services had allowed navigation data service to stabilize and handle all incoming traffic. As a long-term solution, the number of minimum replicas enabled for the navigation data service has been doubled, and additional monitoring alerts have been implemented for this service.