Squiz experienced a major incident on September 16, 2024 affecting Squiz Funnelback Hosted Instances, lasting 3h 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 16, 2024, 08:42 AM UTC
Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers are experiencing slow response times and/or timeouts. We are currently working to find the root cause of this issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
- investigating Sep 16, 2024, 09:01 AM UTC
We are continuing to investigate the situation and our engineers are working to find the root cause of this issue.
- identified Sep 16, 2024, 09:11 AM UTC
Our engineering team has identified an issue with the DXP Funnelback and is in the process of implementing a resolution.
- monitoring Sep 16, 2024, 09:27 AM UTC
We are now beginning to see service improvements in our Funnelback search performance. We'll continue to monitor the situation and look to confirm resolution when possible.
- monitoring Sep 16, 2024, 09:52 AM UTC
We are noting sustained improvements in Funnelback search performance. We will maintain close monitoring of the situation and will provide confirmation of resolution as soon as it is feasible.
- monitoring Sep 16, 2024, 10:41 AM UTC
We are still seeing service improvements in some areas of our Funnelback search performance. We'll continue to monitor the situation and look to confirm resolution when possible.
- monitoring Sep 16, 2024, 11:19 AM UTC
Customers are now confirming that services are recovering and performing as expected. Our team continues to monitor the situation closely to ensure everything remains stable.
- monitoring Sep 16, 2024, 12:23 PM UTC
We continue to monitor the situation and will confirm resolution as soon as possible.
- resolved Sep 16, 2024, 12:36 PM UTC
We are pleased to confirm that the previously reported issue affecting the performance of our UK Funnelback DXP system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.
- postmortem Sep 18, 2024, 03:43 PM UTC
### **Summary** Squiz identified operational issues with Funnelback services in the UK, leading to search function disruptions and latency for several customers. ### **Customer Impact** A small subset of UK Customers may have experienced delays in search results when attempting to utilise the Funnelback search function. ### **Issue and Resolution** We observed a steady increase in traffic, which was able to bypass previous rulesets, by appearing to be from more valid sources. As a result, more targeted and bespoke rules and rate-limiting controls were put in place to prevent systems from being negatively impacted. The traffic these rules targeted were of an automated nature, and no "real person" traffic was impacted by them. We have found no evidence that this was an attack of any sort. In rare circumstances some customers were negatively impacted as a result of these actions. Squiz support teams have liaised with these customers directly to resolve these issues. ### **Mitigation** We have Identified additional protection measures that have now been implemented across our FB DXP Services. * This include enhancing caching rules * Enhanced rate limiting for specific automated traffic sources eg Facebook and Tencent \(WeChat\)