UK Funnelback Performance Degradation

Update timeline

1. investigating Jun 05, 2025, 07:37 AM UTC

   Our monitoring has detected a performance degradation affecting UK Funnelback customers only. We are currently investigating the issue and will provide updates as soon as possible.

2. investigating Jun 05, 2025, 07:43 AM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 05, 2025, 07:50 AM UTC

   Our engineering teams have taken action to restore the service and we are now seeing Funnelback performance returning to normal levels. We will continue to monitor at this time.

4. resolved Jun 05, 2025, 07:59 AM UTC

   Search performance has returned to normal, and the incident is now resolved. We will continue to monitor closely, but no further impact is expected. Thank you for your patience.

5. postmortem Jun 06, 2025, 08:11 AM UTC

   ### Summary On June 5, 2025, Squiz experienced a service degradation affecting UK Funnelback DXP customers. A spike in automated bot traffic led to a strain on system resources, which slowed down search performance and caused occasional service issues for some customers. A major incident was declared at 07:29 UTC. This led to some traffic being temporarily rerouted to alternate endpoints. Mitigation efforts were put in place, including enhanced traffic filtering and bot management measures. Service began to recover at 07:40 UTC following the introduction of targeted blocks and Cloudflare mitigation. The incident was declared resolved after a 10-minute monitoring period, with full service confirmed stable by 07:56 UTC. ### ‌‌Customer Impact Customers in the UK using Funnelback DXP services experienced degraded performance, including slower search responses and intermittent 504 gateway timeout errors. The impact was isolated to the UK region. No disruption was reported in other regions. ### ‌Issue, Resolution, and Mitigation ### Root Cause The incident was caused by a sharp increase in automated traffic, specifically from bot activity targeting UK customers' search services. This spike in non-human traffic led to congestion in the system, triggering rate-limiting protections which also impacted legitimate traffic. ### Resolution Timeline * 07:29 UTC – Major incident declared and incident channel initiated. * 07:32 UTC – A large volume of automated traffic, primarily from a known search engine crawler was observed targeting UK customers’ endpoints.This surge in traffic caused excessive load on the system, triggering rate-limiting protections and impacting legitimate user traffic. * 07:36 UTC – Mitigation initiated & Cloudflare traffic filtering enabled. * 07:39 UTC – Click-ops rule blocks over 3,000 suspicious requests. * 07:40 UTC – Service begins to recover. * 07:56 UTC – Incident declared resolved following successful monitoring. ### Mitigation Actions * Bot traffic was identified and blocked using custom rules. * Rate limits were reviewed and applied selectively. * Cloudflare-based mitigation was enabled to provide immediate relief. * Enhanced monitoring was put in place to verify system recovery.