Nosto experienced a critical incident on November 20, 2023 affecting Onsite products and Integrations and 1 more component, lasting 2h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 20, 2023, 01:22 PM UTC
We have identified an issue with underlying caching system that has caused a wide spread issues in Onsite products, Integrations, API and admin panel.
- identified Nov 20, 2023, 01:57 PM UTC
We are continuing to work on a fix for this issue. The issue is also impacting Search and UGC products.
- monitoring Nov 20, 2023, 01:59 PM UTC
We have provisioned more resources and are monitoring the situation. After adding additional capacity the all systems are operational again.
- monitoring Nov 20, 2023, 02:15 PM UTC
Despite the additional capacity the issue with overloaded caching system reoccured and we're continuing to work on it
- monitoring Nov 20, 2023, 02:17 PM UTC
We're provisioning more resources on the overloaded system and have identified the root cause for the issue. We are preparing a code change that will reduce the load on caching system.
- monitoring Nov 20, 2023, 02:56 PM UTC
We have deployed the code change and the load on caching system is reducing.
- resolved Nov 20, 2023, 03:37 PM UTC
The code change has been deployed and the load has reduced to normal levels. We'll still keep closely monitoring the situation as a precautionary measure.