Nosto Outage History

## **Post-incident summary** During the incident, Nosto Core Personalisation \(including on-site product recommendations and email widgets\) was intermittently slow or unavailable. A sudden traffic spike, largely originating from a single external source, triggered unexpected behavior in the Nosto script’s client-side error reporting. This resulted in a very high volume of error-reporting requests with unusually large payloads being sent to our backend services. The increased load slowed request processing, which then caused additional timeouts and connection errors. Those errors generated further reports, creating a cascading effect where auto-scaling could not keep up with demand. On average, approximately 60–80% of requests \(depending on component\) were impacted during the incident. ### **Impact to other services** Search and Category Merchandising infrastructure remained operational. However, because the Nosto script uses personalisation signals to boost Search and Category Merchandising, some customers experienced failures on category page loads when personalisation calls failed. This affected approximately 24% of Search traffic. Direct GraphQL API calls continued to operate normally. ### **Fixes and prevention** We have completed the post-mortem and implemented the following mitigations: * Updated the Nosto script to handle similar client-side error-reporting scenarios safely and prevent excessive request volume. * Extended rate limiting on the error-reporting endpoint to protect backend services from overload. ### **Ongoing improvements** Work is underway to further reduce risk and improve resilience: * Isolating the logging/error-reporting service from core personalisation services. * Improving alerting and on-call triage processes to detect and mitigate similar patterns faster.