Squiz experienced a major incident on November 25, 2024 affecting Squiz Edge, lasting 5h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 25, 2024, 06:42 AM UTC
Hey Everyone. Squiz has identified issues with the performance of our UK servers and wish to let you know that our engineering team is looking into this matter with top priority and will have updates for you as soon as they are available. Feel free to let us know by our Service Desk if you have any concerns.
- investigating Nov 25, 2024, 06:42 AM UTC
We are continuing to investigate this issue.
- investigating Nov 25, 2024, 07:14 AM UTC
Squiz are continuing to look into the potential causes of the performance degredation at hand and will have an update as soon as pertinent news has arisen.
- monitoring Nov 25, 2024, 07:36 AM UTC
We are now monitoring the performance after our engineering have been able to see some results so far. An update shall be available once we identify no issues.
- identified Nov 25, 2024, 08:52 AM UTC
We have identified this is as a result of DDoS attack which we have partially mitigated, however we are still experiencing issues at this time and our engineers are working to fully mitigate this. Further update to be provided as soon as there is more information available.
- monitoring Nov 25, 2024, 10:56 AM UTC
Following a thorough investigation from our engineering teams, we have performed mitigation and remediation actions. These actions have been successful as we are currently experiencing no significant performance impact. At this time we will continue to monitor the service.
- resolved Nov 25, 2024, 11:53 AM UTC
We are pleased to confirm that the previously reported issue affecting the performance of our UK hosted systems has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.
- postmortem Dec 04, 2024, 03:22 PM UTC
### Summary Squiz identified performance issues with our Compute Service in the UK. ### Customer impact A subset of UK Customers may have experienced delays and timeouts when attempting to access Squiz UK hosted sites. ### Issue and Resolution The root cause of this incident was a DDoS attack. The majority of the attack was automatically blocked by our DDoS protection, but the remainder caused a performance impact to our Compute service in the UK. We performed additional manual intervention to completely mitigate the impact. ### Mitigation In light of this incident, We are investigating a number of approaches in order to reduce the effect of a similar event in the future.