Squiz experienced a major incident on May 13, 2024 affecting Squiz Funnelback Hosted Instances, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 13, 2024, 02:47 PM UTC
Squiz monitoring has detected a degradation of service with Squiz hosted Funnelback. We are working hard to investigate the route cause of the issue and will provide further updates via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.
- investigating May 13, 2024, 03:20 PM UTC
We are continuing to investigate performance issues with Squiz hosted Funnelback in the US. Other regions are unaffected.
- investigating May 13, 2024, 03:44 PM UTC
We are continuing to investigate performance issues with Squiz hosted Funnelback in the US. Our team is actively troubleshooting server issues.
- identified May 13, 2024, 04:00 PM UTC
We have identified issues with search sessions and are taking steps to address the issue.
- monitoring May 13, 2024, 04:21 PM UTC
Changes have been implemented to the session database which should resolve the performance issues. We are currently monitoring to confirm.
- resolved May 13, 2024, 04:34 PM UTC
After identifying issues with sessions and taking steps to repair sessions database the server performance issues have now been resolved.
- postmortem May 13, 2024, 06:09 PM UTC
### Summary During routine monitoring, Squiz identified operational issues with Funnelback services in the US, leading to search function disruptions and latency for several customers. ### Customer impact A subset of US Customers may have experienced delays in search results and encountered 500 errors when attempting to utilise the Funnelback search function. ### Issue and Resolution Squiz engineers were alerted to errors and timeouts originating from our Squiz hosted Funnelback services in the US. This was isolated to the search sessions feature, which was subject to slow response times or termination due a build up of stored requests. In response, we disabled the session storing feature and alleviated the database usage. This will not cause any disruption to search traffic. As part of our standard process we initiated a period of heightened monitoring leading to resolution on May 13th at 17:20 BST ### Mitigation We have added new monitoring checks to flag excess database usage as well as utility scripts to help us debug slow performance in the future. Our Product team is investigating approaches to improve session performance in order to improve overall query performance going forward.