Squiz Outage History

Started May 08, 2025, 12:40 PM UTC· resolved May 08, 2025, 02:07 PM UTC

1. investigating May 08, 2025, 12:40 PM UTC

   Our monitoring has detected an issue resulting in some Funnelback analytics data being unavailable. Our engineers are investigating the issue and updates will be made regularly to this status page.

2. identified May 08, 2025, 12:56 PM UTC

   We have identified the issue and can confirm that it only affects UK customers. We are implementing a rollback to restore analytics services.

3. identified May 08, 2025, 01:32 PM UTC

   We are continuing to investigate the issue and are actively working on a fix.

4. monitoring May 08, 2025, 01:56 PM UTC

   The fix has been implemented and analytics services have been successfully restored. We are currently monitoring the system to ensure continued stability and performance.

5. resolved May 08, 2025, 02:07 PM UTC

   Systems have now recovered, and analytics data has been restored. The incident is resolved.

Read the full incident report →

Started Apr 07, 2025, 03:07 AM UTC· resolved Apr 07, 2025, 03:56 AM UTC

1. investigating Apr 07, 2025, 03:07 AM UTC

   We are currently investigating errors with DXP Funnelback instances where some Australian instances are seeing Funnelback search results impacted. Our Product Development team are looking into this with our engineers to resolve this as soon as possible.

2. investigating Apr 07, 2025, 03:24 AM UTC

   Our Team continue to look into the cause of the impact to Funnelback Performance. We will provide an update once we have any further information on the cause.

3. identified Apr 07, 2025, 03:33 AM UTC

   We are looking into the performance of the affected instances and are looking to implement further resources in order to resolve the issues. We will update with any findings from this action.

4. monitoring Apr 07, 2025, 03:51 AM UTC

   We have implemented some performance fixes and have carried out actions to clean up the processes on the instance affected. We are monitoring to see if the performance of the searches affected are back to normal.

5. resolved Apr 07, 2025, 03:56 AM UTC

   We believe that the issue is now resolved. However we will continue to monitor the affected instances to ensure that the issue does not reoccur. Feel free to raise an incident Via My.Squiz.Net if you believe you have a related issue.

Read the full incident report →

Started Apr 02, 2025, 05:25 AM UTC· resolved Apr 02, 2025, 01:40 PM UTC

1. investigating Apr 02, 2025, 05:25 AM UTC

   Squiz has been made aware of an error affecting customers on the latest version of DXP has caused an error when assets have been placed into safe edit where it will cause an error 500 on trying to access the asset in question. Squiz is working on Rolling back affected clients now.

2. investigating Apr 02, 2025, 05:25 AM UTC

   We are continuing to investigate this issue.

3. identified Apr 02, 2025, 05:38 AM UTC

   Squiz are continuing to work on a Fix for this issue and will implement a fix as soon as we have positive test results.

4. identified Apr 02, 2025, 05:53 AM UTC

   We are now implementing a Rollback for the affected version of DXP. We will have another update ready once the operation is complete.

5. monitoring Apr 02, 2025, 06:18 AM UTC

   We have Begun rolling back clients with the affected version of DXP and have begun monitoring them to see if there are any further issues.

6. monitoring Apr 02, 2025, 06:46 AM UTC

   We are continuing to work on rolling back the changes and are continuing to monitor the situation.

7. monitoring Apr 02, 2025, 07:18 AM UTC

   We are continuing to work on the rollback to resolve the issues.

8. monitoring Apr 02, 2025, 07:36 AM UTC

   Deployment of the rollback is now underway. We will continue to monitor for any issues during the deployment.

9. monitoring Apr 02, 2025, 08:07 AM UTC

   Rollback of the changes continues, and we are now seeing recovery for some affected customers. We are actively checking reported outages for recovery and are continuing to monitor for developments.

10. monitoring Apr 02, 2025, 09:10 AM UTC

    We are continuing to progress with the rollback of the changes. We are continuing to see recovery for some affected customers. We are actively checking reported outages for recovery and are continuing to monitor for developments.

11. monitoring Apr 02, 2025, 10:58 AM UTC

    We are continuing to see recovery for more affected customers and believe we are nearing resolution. We are continuing to be vigilant for any further issues.

12. monitoring Apr 02, 2025, 12:26 PM UTC

    We are continuing to monitor for any outstanding issues. A further update will be provided once the rollback has been completed.

13. resolved Apr 02, 2025, 01:40 PM UTC

    Dear Customers, Following an extended period of monitoring, we are pleased to confirm that this issue has now been resolved. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

14. postmortem Apr 10, 2025, 05:34 AM UTC

    **Summary** On the 2nd April at approximately 14:49 \(GMT\+10\) Squiz received indications of 500 Errors from customer pages. Squiz’s Support teams alongside our product team, quickly identified that the issue was induced by a recent DXP Upgrade version 6.61.0. Hot patches were released whilst in parallel Matrix DXP was rolled back to 6.60.1 restoring services. **Customer Impact** Incident Duration: 02 Apr 2025, 14:49 - 23:14 \(GMT\+10\) Impact: some customers experienced site page 500 errors. Impact times and service restoration times varied throughout the course of the incident duration. The effect of this issue was limited to clients who changed asset statuses during a specific period of time which meant that impact was only felt by some users who would have been editing assets at the time of the incident. **Root cause Analysis** An asset property was removed in Matrix Version 6.61.0. This impacted assets that were placed into Safe Edit as it resulted in errors when Matrix attempted to serialise objects. Resolution Actions 1. Identification Squiz Support Team identified a trend in logs when investigating reports of problems. Product teams were engaged with and quickly isolated the cause. 2. Hot Patch Squiz developed, tested and deployed hot-patches, whilst in parallel assessing version Rollback vs Roll forward. 3. Downgrade To fully resolve the issue a Matrix Version downgrade took place. **Follow-up Actions** Squiz has deployed monitoring enhancements to have the ability to detect/monitor for similar events including identification during testing - completed Squiz has rolled out Matrix Version 6.61.1 successfully, which introduced a change to circumvent this issue. - completed

Read the full incident report →

Started Mar 26, 2025, 11:57 AM UTC· resolved Mar 26, 2025, 04:12 PM UTC

1. investigating Mar 26, 2025, 11:57 AM UTC

   Squiz has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers may experience slow response and or degraded service. We are working to mitigate this problem. More updates to follow shortly. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

2. investigating Mar 26, 2025, 12:08 PM UTC

   We are continuing to investigate this issue.

3. identified Mar 26, 2025, 12:24 PM UTC

   Our engineers have identified a potential source of the issue and are working on implementing a fix.

4. identified Mar 26, 2025, 12:53 PM UTC

   Our engineers are continuing to work through the current incident and are actively testing solutions to resolve the issue.

5. monitoring Mar 26, 2025, 01:20 PM UTC

   We've started to see improvements in search functionality as we continue to monitor the situation.

6. monitoring Mar 26, 2025, 01:55 PM UTC

   Our engineers continue to monitor the situation, as we see signs of recovery.

7. resolved Mar 26, 2025, 04:12 PM UTC

   The issue has been addressed, and services are fully restored. We will continue to monitor system performance, but the incident has now been resolved. A Post-Incident Review (PIR) will be available in the coming days.

8. postmortem Mar 31, 2025, 08:40 AM UTC

   ### Postmortem **Summary** On March 26, 2025, Squiz observed a service degradation affecting several Funnelback DXP customers in the UK. The incident began at 11:57 GMT and lasted for several hours, during which customers experienced slow search responses and degraded service. Engineers identified a potential cause of the issue at 12:24 GMT, and by 13:20 GMT on March 27, signs of recovery were observed. Full service restoration was confirmed by 16:12 GMT, and the issue was resolved. The degradation was primarily caused by unusually high traffic levels coming from a few specific actors, which overwhelmed the system and led to service disruptions. This traffic was subsequently rate-limited to mitigate the impact and restore service functionality. **Customer Impact** A subset of UK customers using Funnelback DXP services experienced slow response times and degraded search functionality. Some users also reported intermittent 504 service errors. The impact was localised to the UK, with no other regions affected. **Issue, Resolution, and Mitigation** Upon investigation, Squiz engineers identified unusually high levels of traffic coming from a small group of specific actors. This high traffic load caused congestion in the system, resulting in delays and intermittent service disruptions for affected customers. The issue was addressed in the following steps: 1. The traffic from the specific actors was identified and rate-limited to reduce the load on the system. 2. Once the rate limiting was applied, service performance began to improve, and the degraded functionality started to recover. 3. Engineers continued to monitor the system for further anomalies and ensured that the rate-limiting measures were effectively restoring normal operations. 4. By 16:12 GMT, the system was fully stabilised, and all affected services were restored.

Read the full incident report →

Started Mar 25, 2025, 10:02 AM UTC· resolved Mar 25, 2025, 04:00 PM UTC

1. investigating Mar 25, 2025, 10:02 AM UTC

   Some customers may see a degradation of performance with connect flows. This is due to a third party provider carrying out essential maintenance. US and EMEA clients service is now performing as normal, as maintenance in this region is complete. AU maintenance will be taking place during the following window. AU Tenant Window: Start: 1:00 PM GMT End: 4:00 PM GMT

2. monitoring Mar 25, 2025, 10:03 AM UTC

   Squiz teams are currently monitoring for any outages caused.

3. monitoring Mar 25, 2025, 10:22 AM UTC

   Our service provider has provided the following details of the US & EMEA outage, we expect similar behaviour during the AU maintenance window. During maintenance, all integration flows were temporarily suspended and resumed after the completion of the upgrades to process all received data. The user interface (frontend) was disabled ensuring a smooth transition and minimal disruption of services.

4. resolved Mar 25, 2025, 04:00 PM UTC

   The maintenance has now concluded and we do not expect any further outages at this time.

Read the full incident report →

Started Feb 21, 2025, 01:22 PM UTC· resolved Feb 21, 2025, 03:24 PM UTC

1. investigating Feb 21, 2025, 01:22 PM UTC

   Squiz monitoring has detected a degradation of service impacted some Funnelback Saas customers in the UK only. Some customer may experience slow response and or degraded service. We are working to mitigate this problem. More updates to follow shortly. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

2. investigating Feb 21, 2025, 01:38 PM UTC

   We are continuing to investigate the issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

3. investigating Feb 21, 2025, 01:55 PM UTC

   Our teams are continuing to investigate the issue at this time. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

4. identified Feb 21, 2025, 02:37 PM UTC

   We believe the issue has been identified and we are implementing a fix, we are starting to see some service level improvements. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

5. identified Feb 21, 2025, 02:46 PM UTC

   Our engineers have found and deployed a solution for this issue, we will continue to monitor for the time being. Performance is now returning to a normal level, further updates will be provided as they are available.

6. monitoring Feb 21, 2025, 03:03 PM UTC

   Our engineers have found and deployed a solution for this issue, we will continue to monitor for the time being. Performance is now returning to a normal level, further updates will be provided as they are available.

7. resolved Feb 21, 2025, 03:24 PM UTC

   We are pleased to confirm that the previously reported issue affecting the performance of our Funnelback system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

Read the full incident report →

Started Jan 30, 2025, 11:14 PM UTC· resolved Jan 30, 2025, 11:55 PM UTC

1. investigating Jan 30, 2025, 11:14 PM UTC

   Squiz has identified an error with Rest resource assets from an update applied overnight and are now investigating a fix for the issue. Feel free to get in contact via MySquiz if you believe you are affected.

2. identified Jan 30, 2025, 11:23 PM UTC

   Our Team have found a Patch for this issue which has worked on a test instance. We are now working on applying this to a wider group.

3. monitoring Jan 30, 2025, 11:47 PM UTC

   The patch mentioned in our previous update has now been applied across production. We will continue to monitor to see if this has been resolved.

4. resolved Jan 30, 2025, 11:55 PM UTC

   This Issue has now been considered resolved after our observations. Feel free to get in contact by MySquiz.net to let us know if you have any questions.

Read the full incident report →

Started Jan 14, 2025, 03:03 PM UTC· resolved Jan 14, 2025, 04:14 PM UTC

1. investigating Jan 14, 2025, 03:03 PM UTC

   We are currently witnessing intermittent network issues with one of our transit providers in the AU. We’ll continue to investigate the situation and provides updates as soon as possible.

2. investigating Jan 14, 2025, 03:03 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jan 14, 2025, 03:40 PM UTC

   A fix has been implemented and services are restored. We are currently still monitoring the situation.

4. resolved Jan 14, 2025, 04:14 PM UTC

   This incident has been resolved and all services are up and running

5. postmortem Jan 15, 2025, 02:44 PM UTC

   **Summary** Several AU customers experienced website degradation between Jan 14, 2025, 23:47 AEST - Jan 15, 2025, 1:40 AEST. Squiz identified operational issues with one of our third party network providers. This had a negative effect on Matrix and Funnelback services in the AU, leading to search function disruptions, latency for several customers and some 504 errors. On January the 15th, at 19:51 AEST, during a scheduled maintenance window there was a further observed degradation of service. This was a re-occurrence of the same issue. ### **Customer Impact** A small subset of AU Customers experienced delays in search results when attempting to utilise the Funnelback search and web functions, as well as some more 504 service outages. ### **Issue, Resolution and Mitigation** We experienced concurrent intermittent traffic loss to/from NTT in both our Sydney and Melbourne DCs. The traffic loss was severe enough to trigger automatic rerouting of Ingress traffic to a different Transit provider. Because the packet loss was intermittent, this rerouting process resolved, then repeated several times. We intervened manually to force the exclusive use of a different Transit Provider in Sydney. This partially mitigated the issue, but it took some time for routing to fully recover. This also affected some of our internal observability systems. Once the NTT transit was stable, we reverted our mitigations to restore full redundant service.

Read the full incident report →

Started Dec 17, 2024, 09:57 AM UTC· resolved Dec 17, 2024, 10:13 AM UTC

1. resolved Dec 17, 2024, 02:36 PM UTC

   09:57-10:13am GMT Excess traffic towards the UK DXP Search Admin API caused a denial of service situation where some users received 504 Gateway Timeouts. Service was restored by cycling services and reducing traffic.

Read the full incident report →

Started Nov 25, 2024, 06:42 AM UTC· resolved Nov 25, 2024, 11:53 AM UTC

1. investigating Nov 25, 2024, 06:42 AM UTC

   Hey Everyone. Squiz has identified issues with the performance of our UK servers and wish to let you know that our engineering team is looking into this matter with top priority and will have updates for you as soon as they are available. Feel free to let us know by our Service Desk if you have any concerns.

2. investigating Nov 25, 2024, 06:42 AM UTC

   We are continuing to investigate this issue.

3. investigating Nov 25, 2024, 07:14 AM UTC

   Squiz are continuing to look into the potential causes of the performance degredation at hand and will have an update as soon as pertinent news has arisen.

4. monitoring Nov 25, 2024, 07:36 AM UTC

   We are now monitoring the performance after our engineering have been able to see some results so far. An update shall be available once we identify no issues.

5. identified Nov 25, 2024, 08:52 AM UTC

   We have identified this is as a result of DDoS attack which we have partially mitigated, however we are still experiencing issues at this time and our engineers are working to fully mitigate this. Further update to be provided as soon as there is more information available.

6. monitoring Nov 25, 2024, 10:56 AM UTC

   Following a thorough investigation from our engineering teams, we have performed mitigation and remediation actions. These actions have been successful as we are currently experiencing no significant performance impact. At this time we will continue to monitor the service.

7. resolved Nov 25, 2024, 11:53 AM UTC

   We are pleased to confirm that the previously reported issue affecting the performance of our UK hosted systems has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

8. postmortem Dec 04, 2024, 03:22 PM UTC

   ### Summary Squiz identified performance issues with our Compute Service in the UK. ### Customer impact A subset of UK Customers may have experienced delays and timeouts when attempting to access Squiz UK hosted sites. ### Issue and Resolution The root cause of this incident was a DDoS attack. The majority of the attack was automatically blocked by our DDoS protection, but the remainder caused a performance impact to our Compute service in the UK. We performed additional manual intervention to completely mitigate the impact. ### Mitigation In light of this incident, We are investigating a number of approaches in order to reduce the effect of a similar event in the future.

Read the full incident report →

Started Oct 28, 2024, 03:05 AM UTC· resolved Oct 28, 2024, 11:44 AM UTC

1. investigating Oct 28, 2024, 03:05 AM UTC

   Our team is actively working on the SSL certificate renewal Certificate update process is in progress

2. identified Oct 28, 2024, 03:24 AM UTC

   Our team continue to work on the SSL certificate renewal

3. identified Oct 28, 2024, 09:21 AM UTC

   Our network teams are engaged on resolving this issue. We will provide further updates as more information becomes available.

4. monitoring Oct 28, 2024, 09:41 AM UTC

   The SSL certificate has now been renewed and Connect in AU is working as expected. We will continue to monitor for any lasting issues.

5. resolved Oct 28, 2024, 11:44 AM UTC

   This issue has now been resolved. We have retested Connect AU and the UI errors are gone and the service is working as expected.

Read the full incident report →