Squiz Outage History

Started Sep 10, 2026, 11:26 AM UTC· resolved Sep 10, 2026, 02:29 PM UTC

1. investigating Sep 10, 2026, 11:26 AM UTC

   We are currently seeing performance issues on for UK based Funnelback clients. Our technical teams are currently investigating the issue.

2. investigating Sep 10, 2026, 12:05 PM UTC

   Teams are continuing to investigate the cause of this slowness.

3. monitoring Sep 10, 2026, 12:35 PM UTC

   A fix has implemented and we are monitoring results

4. resolved Sep 10, 2026, 02:29 PM UTC

   This incident has been resolved. Information pertaining to this incident will be available via this status page in the near future.

Read the full incident report →

Started Sep 10, 2026, 01:58 AM UTC· resolved Sep 10, 2026, 05:21 AM UTC

1. identified Sep 10, 2026, 01:58 AM UTC

   Dear Customer, Search (Funnelback) services in the ANZ region are undergoing brief emergency maintenance between 1:30PM AEST and 3:30PM AEST. We are currently experiencing an unexpected increase in task queue times for a subset of ANZ Search (Funnelback) customers. A minor administrative access delay of under two minutes may occur during this maintenance timeframe. Date: 10th September 2026 Maintenance Window Start: 1:30PM AEST Maintenance Window End: 3:30PM AEST Expected Duration: up to 2 minutes Impact: A minor administrative access delay of under two minutes may occur during the maintenance window.

2. monitoring Sep 10, 2026, 03:38 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 10, 2026, 05:21 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 16, 2026, 12:50 PM UTC· resolved Jun 17, 2026, 02:36 PM UTC

1. investigating Jun 16, 2026, 12:50 PM UTC

   We are aware of higher than normal error rates for a subset of EMEA Funnelback clients. We are currently investigating.

2. monitoring Jun 16, 2026, 11:10 PM UTC

   A fix was implemented immediately when this issue was raised - however this statuspage failed to update. There is no customer impact, and we are still monitoring the situation closely

3. resolved Jun 17, 2026, 02:36 PM UTC

   This incident has been resolved.

4. postmortem Jun 19, 2026, 12:34 PM UTC

   **Summary** A spike in crawling traffic originating from Facebook \(Meta\) targeted search pages, caused excessive load on the **DXP search infrastructure**. Initial mitigation involved blocking the crawler at the individual account level. Upon identifying that the traffic was dynamically shifting across multiple accounts over time, a **global rule** was implemented to block the traffic across all search instances, successfully stabilizing the platform. **Customer Impact** A subset of **UK-based clients** experienced intermittent intermittent timeout / HTTP 500 errors when attempting to access their infrastructure during the period of elevated load. **Root Cause** The issue was caused by elevated crawling activity originating from Facebook infrastructure that targeted specific search pages, leading to temporary resource constraints on the **DXP search infrastructure**. **Resolution Actions** * **Proactive Detection:** Internal monitoring systems successfully flagged the traffic spike and elevated resource usage * **Targeted Mitigation:** Implemented localized blocking rules on the specific client accounts initially affected by the crawling activity. * **Global Rule Implementation:** Deployed a global rule across all search instances to comprehensively block the crawler traffic, fully restoring and stabilizing platform performance. * **Long-Term Prevention:** Squiz is currently evaluating origin-agnostic bot mitigation strategies to comprehensively protect search infrastructure from aggressive or automated crawling activity, regardless of its source.

Read the full incident report →

Started Jun 16, 2026, 09:56 AM UTC· resolved Jun 16, 2026, 10:24 AM UTC

1. investigating Jun 16, 2026, 09:56 AM UTC

   Between approximately 08:15 and 09:15 BST today we were aware of higher than normal error rates for a subset of EMEA Funnelback clients. Our engineers have isolated the issue and have taken corrective action.

2. monitoring Jun 16, 2026, 09:56 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 16, 2026, 10:24 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 12, 2026, 11:50 PM UTC· resolved Mar 13, 2026, 01:03 AM UTC

1. investigating Mar 12, 2026, 11:50 PM UTC

   Customers are currently experiencing issues with accessing Advanced Forms in the DXP via the Console. Users attempting to access the service are observing HTTP502 errors and/or login forms. The incident team is currently investigating the cause.

2. identified Mar 13, 2026, 12:15 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 13, 2026, 12:42 AM UTC

   A fix has been implemented and we have started to observe recovery. We are monitoring the roll out of the recovery and will confirm that operations are returned to normal once the process has completed.

4. resolved Mar 13, 2026, 01:03 AM UTC

   The incident has been resolved and Advanced Forms access should be restored for all customers.

Read the full incident report →

Started Mar 04, 2026, 03:52 PM UTC· resolved Mar 04, 2026, 10:57 PM UTC

1. investigating Mar 04, 2026, 03:52 PM UTC

   We are currently investigating an issue affecting Connect Ordinary Flows on our US instance. Due to a service disruption with our upstream cloud provider in the US East (Ashburn) region, new containers are currently unable to launch. Please note that all incoming messages are being securely held in our queues. No data is being lost, and processing will resume automatically once the connection is restored. Our engineering team is working closely with the provider to expedite a resolution. We will provide further updates as more information becomes available.

2. investigating Mar 04, 2026, 04:25 PM UTC

   We are continuing to work with the provider on full resolution for all affected services. We will provide updates as they become available.

3. investigating Mar 04, 2026, 04:55 PM UTC

   We are continuing to work with the provider for a resolution. We will provide updates as they become available.

4. investigating Mar 04, 2026, 05:30 PM UTC

   We are continuing to work with the provider on full resolution for all affected services. We will provide updates as they become available.

5. identified Mar 04, 2026, 05:57 PM UTC

   We are seeing partial restoration at this time and are monitoring closely.

6. monitoring Mar 04, 2026, 06:36 PM UTC

   Ordinary Flows are beginning to operate now. We continue to monitor the progress closely while working to full resolution.

7. monitoring Mar 04, 2026, 07:03 PM UTC

   Our engineering team continues to monitor and work with the provider for resolution. We will continue to provide updates as more information becomes available.

8. monitoring Mar 04, 2026, 07:33 PM UTC

   We are continuing the monitoring, but seeing better results. We will provide further updates as more information becomes available.

9. monitoring Mar 04, 2026, 08:00 PM UTC

   We are continuing the monitoring, but seeing better results. We will provide further updates as more information becomes available.

10. monitoring Mar 04, 2026, 08:30 PM UTC

    We are continuing the monitoring, but seeing better results. We will provide further updates as more information becomes available.

11. monitoring Mar 04, 2026, 09:33 PM UTC

    We are continuing to work with our provider for full resolution. We will provide further updates as more information becomes available.

12. monitoring Mar 04, 2026, 10:30 PM UTC

    Our vendor has indicated that the platform should now be fully operational. We will continue to monitor to confirm resolution, and provide updates as more details become available.

13. resolved Mar 04, 2026, 10:57 PM UTC

    Squiz teams have worked with our 3rd party service provider, and a fix has been deployed for the current Incident. This restored service for all affected customers. We apologise for this degradation of service and thank you for your patience while we worked on the resolution. A postmortem will be provided via https://status.squiz.cloud once we have worked with our service provider.

Read the full incident report →

Started Feb 20, 2026, 06:09 PM UTC· resolved Feb 21, 2026, 12:00 AM UTC

1. investigating Feb 20, 2026, 06:09 PM UTC

   We are currently investigating an issue impacting all our services. Initial diagnostics suggest a widespread disruption with Cloudflare, our content delivery provider. Our team is monitoring the situation closely and will provide updates as they become available.

2. identified Feb 20, 2026, 06:29 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 20, 2026, 06:30 PM UTC

   A fix has been implemented and we are monitoring the results.

4. investigating Feb 20, 2026, 06:37 PM UTC

   We are investigating additional issues for all affected clients. We will provide further updates as they become available.

5. identified Feb 20, 2026, 07:03 PM UTC

   We are working with Cloudflare regarding connectivity issues. The Cloudflare status page can be found here: https://www.cloudflarestatus.com/incidents/kwy3dt82bwbt

6. identified Feb 20, 2026, 07:24 PM UTC

   We continue to work with Cloudflare regarding the ongoing issues. We will provide further updates as they become available.

7. identified Feb 20, 2026, 07:53 PM UTC

   We are continuing to work on full resolution with Cloudflare. We will provide further updates as they become available.

8. identified Feb 20, 2026, 08:15 PM UTC

   We are continuing to work with our content delivery provider for full resolution. We will provide updates as they become available.

9. identified Feb 20, 2026, 08:45 PM UTC

   We are continuing to work on full resolution and will provide updates as they become available.

10. identified Feb 20, 2026, 09:17 PM UTC

    We are continuing to work with Cloudflare on full resolution for all affected services. We will provide updates as they become available.

11. identified Feb 20, 2026, 09:22 PM UTC

    We are continuing to work with Cloudflare on full resolution for all affected services. We will provide updates as they become available.

12. identified Feb 20, 2026, 09:54 PM UTC

    We are continuing to work with Cloudflare on full resolution for all affected services. We will provide updates as they become available.

13. identified Feb 20, 2026, 10:05 PM UTC

    We are seeing recovery of services at this time and we are monitoring the situation to ensure things remain stable.

14. monitoring Feb 20, 2026, 10:05 PM UTC

    We are seeing recovery of services at this time and we are monitoring the situation to ensure things remain stable.

15. monitoring Feb 20, 2026, 10:33 PM UTC

    We are seeing recovery of services at this time and we are continuing to monitoring the situation to ensure all services remain stable.

16. monitoring Feb 20, 2026, 11:02 PM UTC

    We are still monitoring this situation at large as well as investigating a handful of remaining intermittent issues. We will provide additional updates as available.

17. monitoring Feb 20, 2026, 11:30 PM UTC

    We are still seeing recovery of services at this time and we are continuing to monitoring the situation to ensure all services remain stable.

18. resolved Feb 21, 2026, 12:00 AM UTC

    This incident has been resolved.

19. postmortem Feb 23, 2026, 03:54 AM UTC

    **Executive Summary** On Friday, February 20, 2026, at 1748 UTC, a global Cloudflare incident impacted network routing used to deliver traffic to services hosted on the Squiz DXP platform. During this time: * All DXP users, and Squiz cloud customers using Cloudflare, were unable to log in to any administrative interfaces in the Squiz DXP, including the Console, Content Management System \(Matrix\), and Funnelback. * Customer frontend websites using Cloudflare were returning cloudflare errors “Connection timed Out” This issue was caused by a change that Cloudflare made to how our network manages IP addresses. Cloudflare have provided a detailed analysis of the incident at [https://blog.cloudflare.com/cloudflare-outage-february-20-2026/](https://blog.cloudflare.com/cloudflare-outage-february-20-2026/) Restoration of the cloudflare service for individual customers ranged from between 30 mins to 3.5 hours depending on the IP configurations for each customer. If you require more information specific to your solution please log a ticket via the My.Squiz portal requesting a Custom Post Incident Report. Squiz would like to remind customers to subscribe to the Squiz Status page at [status.squiz.net](http://status.squiz.net) to receive real time notification of Major Incidents. **References** * Cloudflare incident Page: [https://www.cloudflarestatus.com/incidents/kwy3dt82bwbt](https://www.cloudflarestatus.com/incidents/kwy3dt82bwbt) * Cloudflare incident Blog: [https://blog.cloudflare.com/cloudflare-outage-february-20-2026/](https://blog.cloudflare.com/cloudflare-outage-february-20-2026/) * Squiz Status Page: [https://status.squiz.cloud/](https://status.squiz.cloud/) **Timeline: 20 Feb \(UTC\)** * Incident start time: 1749 UTC * Squiz Incident declaration: 1800 UTC * Initial Squiz Status Page notifications posted: 1809 UTC * Cloudflare reported full resolution for anycast customers: 2053 UTC * Cloudflare reported full resolution for spectrum customers: 2259 UTC * Cloudflare marked the incident resolved on at 23:50 UTC * Squiz Declared Incident resolved: 0000 21 Feb UTC

Read the full incident report →

Started Jan 23, 2026, 12:59 AM UTC· resolved Jan 24, 2026, 12:28 PM UTC

1. investigating Jan 23, 2026, 03:22 AM UTC

   We're investigating a possible issue with one of our 3rd party network providers that's affecting some clients in the Squiz Cloud. We'll let you know when we have more details.

2. investigating Jan 23, 2026, 03:51 AM UTC

   We are continuing to work with our upstream vendors on this issue

3. investigating Jan 23, 2026, 05:47 AM UTC

   Services remain affected. We are continuing to work to restore affected services.

4. investigating Jan 23, 2026, 06:04 AM UTC

   The fault has been isolated to an upstream issue with a network provider. We are working with vendors on a resolution, and will update as soon as we have more information.

5. investigating Jan 23, 2026, 07:13 AM UTC

   Services remain affected. We are continuing to work to restore affected services.

6. investigating Jan 23, 2026, 09:51 AM UTC

   We are continuing to work with our upstream providers, who have identified a potential root cause for this issue. We will provide further updates when we have more information.

7. investigating Jan 23, 2026, 03:21 PM UTC

   We are continuing investigation into this issue with our upstream providers. No definitive root cause at this stage.

8. investigating Jan 23, 2026, 07:14 PM UTC

   We are continuing investigation into this issue with our upstream providers. No definitive root cause at this stage.

9. investigating Jan 23, 2026, 11:14 PM UTC

   We are continuing investigation into this issue with our upstream providers. No definitive root cause at this stage.

10. monitoring Jan 24, 2026, 02:40 AM UTC

    We are seeing recovery of affected services. We will continue monitoring to ensure the issue does not reoccur.

11. monitoring Jan 24, 2026, 12:23 PM UTC

    This incident is now resolved, we have been monitoring closely, and have not seen any further issues.

12. resolved Jan 24, 2026, 12:28 PM UTC

    This incident has been resolved.

13. postmortem Jan 30, 2026, 08:42 AM UTC

    ## Incident Summary A problem on a 3rd party Network impacted the Internet connectivity between two different service providers that Squiz utilise for Internet transit and CDN for some customers hosted within our Australian Data Centres. Squiz DXP services were unaffected The service degradation began on **Jan 23, 11:56 AEDT**, and ended **Jan 24, 10:22 AEDT** when the 3rd party Network operator fixed their network. Squiz continued to monitor for any further impact for an extended length of time, and the incident was officially resolved **Jan 24, 23:26 AEDT,** once we received confirmation from our service provider as to the root cause. ‌ ## Incident Timeline This is a high-level summary of the key events during the incident. All times are in AEDT. | **Time** | **Event** | | --- | --- | | Jan 23, 2026 | | | 11:56 | Service Degradation begins. | | 12:33 | First report of customer impact. | | 12:34 onwards | Further reports of impacted services, and initial investigation to identify the cause. | | 14:17 | A formal incident was declared and a response team assembled to investigate. | | 15:49 | Following multiple attempted mitigation efforts by Squiz Technical Staff, and further analysis, the problem was escalated to Squiz’s Internet Transit and CDN service providers. | | 19:24 | Squiz’s Service providers identify an issue with connectivity between each other, via a 3rd party network. | | Jan 24, 2026 | | | 09:02 | Signs of substantial service improvement. | | 10:22 | Service degradation ends and monitoring begins. | | 19:40 | Confirmation received from our transit service provider that the 3rd party network operator implemented a fix at the time we saw services return to normal. | | 23:26 | Incident Resolved | ## Customer Impact A subset of Squiz customers with sites hosted within those Data Centres, \(those whose service was dependent on both these service providers\) experienced mild to severe service degradation throughout this incident - typically manifesting as pages failing to load, excessive load times, and an inability to access administration interfaces. As a result, this incident was classified as a Priority 1. DXP services were unaffected. ## Root Cause Analysis The incident was caused by a problem on a 3rd party Network between two different service providers that Squiz utilise for Internet transit and CDN for some customers hosted within our Australian Data Centres.The issue occurred outside our platform and network boundary, and that of all our immediate partners and service providers. ## Mitigation and Corrective Actions No technical corrective actions were noted for Squiz, as the incident was due to a problem with a third-party network with whom Squiz does not have a relationship.

Read the full incident report →

Started Dec 05, 2025, 09:17 AM UTC· resolved Dec 05, 2025, 09:36 AM UTC

1. monitoring Dec 05, 2025, 09:17 AM UTC

   We are currently experiencing degraded performance for all clients using Cloudflare-backed services

2. monitoring Dec 05, 2025, 09:18 AM UTC

   Cloudflare has announced that they have implemented a fixed and we are seeing recovery on all our services.

3. resolved Dec 05, 2025, 09:36 AM UTC

   Cloudflare has confirmed that the issue is resolved, and we are also declaring it resolved on our end. For further details, please refer back to this page.

Read the full incident report →

Started Nov 20, 2025, 10:35 PM UTC· resolved Nov 20, 2025, 11:22 PM UTC

1. investigating Nov 20, 2025, 10:35 PM UTC

   We are currently investigating an issue affecting authentication for the Funnelback Search Admin console on POD01 (Australia). Impact: This incident is currently affecting users attempting to access https://dxp-au-admin.funnelback.squiz.cloud/. Affected Roles: Users assigned the DXP Developer role. Extended Impact: This also affects users with higher-level privileges that include Developer permissions, specifically Site Admin roles and above. Users may experience errors or be unable to log in to the Funnelback console. Standard end-user search functionality on your public websites remains operational; this is limited to the Admin interface. We are actively investigating the root cause and will provide an update shortly.

2. identified Nov 20, 2025, 11:03 PM UTC

   Following further investigation into the access issues on POD01 (Australia), we have confirmed that general logins and access to the Funnelback Admin console (https://dxp-au-admin.funnelback.squiz.cloud/) are operational for the majority of clients. Current Observations: Login Access: The inability to log in appears to be isolated to a specific subset of client environments. We are currently working directly with the affected accounts. Feature Availability: While most clients can successfully log in, users may experience intermittent issues with specific capabilities or features within the DXP Developer and Site Admin views while we complete our remediation.

3. resolved Nov 20, 2025, 11:22 PM UTC

   We have completed our investigation into the reported access and permissions issues affecting the Funnelback Admin console on POD01 (Australia). Our engineering team has performed comprehensive checks on the authentication infrastructure. We have confirmed that Single Sign-On (SSO) services and DXP User Role assignments are functioning as intended. No platform-wide remediation was required. All users with the correct role assignments (DXP Developer, Site Admin, and above) can access the console as normal.

Read the full incident report →

Started Nov 18, 2025, 12:13 PM UTC· resolved Nov 18, 2025, 07:50 PM UTC

1. investigating Nov 18, 2025, 12:13 PM UTC

   We are currently investigating the issue.

2. investigating Nov 18, 2025, 12:42 PM UTC

   We are continuing to investigate this issue.

3. identified Nov 18, 2025, 02:27 PM UTC

   We are seeing recovery across many sites but we are seeing periodic issues still. We’re monitoring the situation and are in communication with Cloudflare.

4. monitoring Nov 18, 2025, 02:47 PM UTC

   A fix has been implemented and we are now monitoring the results.

5. monitoring Nov 18, 2025, 03:54 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Nov 18, 2025, 07:50 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 26, 2025, 01:16 AM UTC· resolved Oct 27, 2025, 06:39 AM UTC

1. investigating Oct 26, 2025, 01:16 AM UTC

   We are experiencing issues with our customer support services. Support calls may not be getting through and customers may be having issues submitting support tickets through the MySquiz portal. For any urgent issues, please email [email protected].

2. identified Oct 26, 2025, 11:39 PM UTC

   MySquiz can now be used, any issues please let us know at [email protected]. Phone system is still having issues and we are working to resolve it as soon as possible

3. resolved Oct 27, 2025, 06:39 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 07:29 AM UTC· resolved Oct 20, 2025, 11:16 PM UTC

1. investigating Oct 20, 2025, 07:29 AM UTC

   We’re currently experiencing issues when trying to log-in to DXP systems. Our engineers are investigating the issue and will provide updates via the Squiz Status page.

2. investigating Oct 20, 2025, 07:29 AM UTC

   We are continuing to investigate this issue.

3. investigating Oct 20, 2025, 07:45 AM UTC

   Our Engineers are continuing to investigate the issue at hand and will provide more details once more is known.

4. investigating Oct 20, 2025, 08:06 AM UTC

   AWS is currently seeing critical issues, see details here. The following Squiz services are impacted: Global: - DXP Authentication - Static sites served from Filestore US Region only: - Component Service - Page Builder - Conversational Search

5. investigating Oct 20, 2025, 08:37 AM UTC

   AWS still currently seeing critical issues, see details here: https://health.aws.amazon.com/health/status The following Squiz services are impacted: Global: - DXP Authentication - Static sites served from Filestore - Squiz Support Services - MySquiz/Jira - Squiz Support phone lines - Squiz Status Page US Region only: - Component Service - Page Builder - Conversational Search

6. identified Oct 20, 2025, 08:37 AM UTC

   The issue has been identified and a fix is being implemented.

7. identified Oct 20, 2025, 08:37 AM UTC

   We are continuing to work on a fix for this issue.

8. identified Oct 20, 2025, 09:26 AM UTC

   AWS has applied mitigations and we are starting to see recovery on Squiz services

9. monitoring Oct 20, 2025, 09:52 AM UTC

   AWS is reporting significant signs of recovery. Squiz is also now seeing recovery on all previously listed services. We will continue to monitor the situation until AWS marks the incident as resolved.

10. monitoring Oct 20, 2025, 01:10 PM UTC

    Squiz is seeing all of our services as stable and fully recovered. We will however keep this incident open until Amazon declares full recovery. We are continuing to closely monitor the situation

11. monitoring Oct 20, 2025, 04:34 PM UTC

    We are continuing to monitor for any further issues.

12. monitoring Oct 20, 2025, 06:35 PM UTC

    We are continuing to monitor for any further issues.

13. monitoring Oct 20, 2025, 07:56 PM UTC

    AWS is reporting significant signs of recovery but this issue is still on-going. We will continue to monitor the situation until AWS marks the incident as resolved.

14. monitoring Oct 20, 2025, 10:28 PM UTC

    We are continuing to Monitor for any changes in the status of this issue.

15. resolved Oct 20, 2025, 11:16 PM UTC

    We have received confirmation from AWS that the incident shall now be resolved. We will continue to monitor for changes and will communicate if a further issue arises. Feel free to let us know at My.Squiz.Net if you believe you are still affected.

Read the full incident report →

Started Oct 20, 2025, 05:20 AM UTC· resolved Oct 20, 2025, 06:12 AM UTC

1. investigating Oct 20, 2025, 05:20 AM UTC

   Dear Valued Customers. We have come across an issue affecting the component services within DXP which is affecting the creation of new components using the DXP Command Line Interface. We are only noticing this in the APAC region. Our Engineers are working towards a fix to be applied as soon as possible.

2. identified Oct 20, 2025, 05:33 AM UTC

   We have Identified the cause of this issue and are now looking into potential fixes to be applied to bring this service back into working order.

3. identified Oct 20, 2025, 05:54 AM UTC

   Our Engineers are continuing to investigate a fix for these issues at hand to ensure that it will cover the whole issue and not lead to side affects.

4. identified Oct 20, 2025, 06:12 AM UTC

   Our Engineers have now considered this issue now resolved. However we will remain vigilant incase of any further issues. If you believe something is still not right related to this incident then feel free to contact us on My.Squiz.Net

5. resolved Oct 20, 2025, 06:12 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 17, 2025, 08:49 AM UTC· resolved Oct 17, 2025, 09:02 AM UTC

1. investigating Oct 17, 2025, 08:49 AM UTC

   We are currently investigating an issue with our Squiz Cloud UK DC.

2. identified Oct 17, 2025, 08:54 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 17, 2025, 08:54 AM UTC

   A fix has been implemented and we are monitoring the results

4. resolved Oct 17, 2025, 09:02 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 15, 2025, 03:38 PM UTC· resolved Oct 15, 2025, 03:54 PM UTC

1. investigating Oct 15, 2025, 03:38 PM UTC

   We are currently investigating an issue affecting Funnelback services for some customers in our UK datacenter

2. monitoring Oct 15, 2025, 03:46 PM UTC

   We have identified the source of the issue and have implemented a fix and are currently monitoring the situation

3. resolved Oct 15, 2025, 03:54 PM UTC

   This incident has been resolved

Read the full incident report →

Started Sep 10, 2025, 04:25 PM UTC· resolved Sep 10, 2025, 07:35 PM UTC

1. investigating Sep 10, 2025, 04:25 PM UTC

   We are currently investigating an issue which is causing degraded performance in our ANZ Squiz Cloud Datacenter, affected systems may be experiencing slow load times or intermittent outages. Squiz teams are looking into this now and we will provide updates as they are available.

2. investigating Sep 10, 2025, 04:51 PM UTC

   Squiz teams are continuing to investigate this issue, we will provide more information as it becomes available

3. identified Sep 10, 2025, 05:25 PM UTC

   The issue has been identified and mitigation steps are taking place to resolve the matter

4. monitoring Sep 10, 2025, 05:55 PM UTC

   A fix has been implemented and we are monitoring the affected datacenter to ensure the issue remains resolved

5. resolved Sep 10, 2025, 07:35 PM UTC

   This incident is resolved.

Read the full incident report →

Started Aug 07, 2025, 08:15 PM UTC· resolved Aug 07, 2025, 09:00 PM UTC

1. investigating Aug 07, 2025, 08:15 PM UTC

   We are currently investigating server issues on the Funnelback legacy environment.

2. identified Aug 07, 2025, 08:37 PM UTC

   We have identified source of the degradation and are taking steps to resolve

3. monitoring Aug 07, 2025, 08:58 PM UTC

   We have taken steps to resolve the issue causing the degradation. All server processes have returned to normal. We are continuing to monitor.

4. resolved Aug 07, 2025, 09:00 PM UTC

   We have taken steps to resolve the issue causing the degradation. All server processes have returned to normal.

Read the full incident report →

Started Jul 22, 2025, 11:24 PM UTC· resolved Jul 23, 2025, 12:27 AM UTC

1. investigating Jul 22, 2025, 11:24 PM UTC

   We are currently Investigating a section of our DXP clients who are experiencing timeouts when Funnelback is used. We will post more updates as the investigation progresses.

2. monitoring Jul 22, 2025, 11:36 PM UTC

   We have made some changes to the configuration of the hosting environment in order to try and get more robust performance out of the hosted Funnelback Instances. We will continue monitoring to see if any further changes are required.

3. monitoring Jul 23, 2025, 12:01 AM UTC

   We are continuing to see positive performance for the affected instances in this issue. Our team are continuing to Monitor for issues and any improvements to be made.

4. resolved Jul 23, 2025, 12:27 AM UTC

   We can confirm that the performance of search instances are now stable and we are considering this incident now closed. If you have any lingering issues or have concerns then please feel free to get in touch via My.Squiz.Net

Read the full incident report →

Started Jul 22, 2025, 12:34 AM UTC· resolved Jul 22, 2025, 01:34 AM UTC

1. investigating Jul 22, 2025, 12:34 AM UTC

   We have noticed that the Newbury Datacentre in the UK is currently offline. This may affect Disaster recovery and development instances in the UK region as other parts of our infrastructure will take on the load. We are currently investigating the cause of the error and will post more updates as this progresses.

2. investigating Jul 22, 2025, 12:43 AM UTC

   We are continuing to investigate this issue.

3. identified Jul 22, 2025, 01:15 AM UTC

   We are working with our Network provider to work on the issue at hand. More updates to be posted once we have any more information to share.

4. resolved Jul 22, 2025, 01:34 AM UTC

   We can now confirm that performance of the Data Center is now back to a running state. If you believe you have any issues lingering that are related then feel free to get in contact via My.Squiz.Net.

Read the full incident report →

Started Jul 21, 2025, 09:52 AM UTC· resolved Jul 21, 2025, 10:21 AM UTC

1. investigating Jul 21, 2025, 09:52 AM UTC

   We are currently experiencing issues with connect functionality and are experiencing delays in processing and starting executions. Please bear with us while we investigate the issue.

2. resolved Jul 21, 2025, 10:21 AM UTC

   We are no longer seeing issues with Connect, and performance has returned to the usual level. Please get in touch via My.Squiz.Net to let us know if you have any issues you believe are related. We will also provide a post mortem for this incident in the next few days.

Read the full incident report →

Started Jul 16, 2025, 03:03 AM UTC· resolved Jul 16, 2025, 03:55 AM UTC

1. investigating Jul 16, 2025, 03:03 AM UTC

   Hello Customers. We have an incident raised now regarding configurations reverting to old versions and failures in pushing new collections for clients who have Funnelback hosted in Australia and in a SAAS/Cloud infrastructure. Our team are now working on rectifying Funnelback functionality.

2. identified Jul 16, 2025, 03:28 AM UTC

   We have identified the cause of the issue at hand and are now on our way to producing a fix for this. If you believe you are affected then feel free to log a case with us on My.Squiz.Net

3. monitoring Jul 16, 2025, 03:47 AM UTC

   We have rectified the issue and are now trying to identify any outstanding items left to wrap up this incident. If you believe you are affected you should now be able to repush your collections now to rectify your instance.

4. resolved Jul 16, 2025, 03:55 AM UTC

   This incident is now declared as resolved. Our team have identified the clients who need to be contacted in order to push their collections again. Feel free to let us know if you have any outstanding issues via My.Squiz.Net

5. postmortem Jul 30, 2025, 03:59 AM UTC

   **Summary** **On 16 July 2025 at approximately 13:00 AEST, Squiz observed an issue affecting our Funnelback AU platform.Some customers’ search data appeared to have reverted to an earlier state \(30 June 2025\), impacting the accuracy of their search results.** **This was triggered by an admin server restart that inadvertently caused an outdated recovery volume to be mounted.While this older volume was not intended to be active, a technical conflict led it to override the correct data volume mount during startup.** **Squiz engineers took corrective action by detaching the incorrect volume and restarting the affected system.Full service was restored and verified by 13:54 AEST the same day.** ‌ ### **Impact** **Customers hosted on the affected Pod 01 may have experienced outdated or missing search data. This included:** * **Reverted content in admin server screens/searches - live search was unaffected** * **Delays in document updates being reflected in search** * **Temporary inconsistencies between backend systems and frontend content** **While the issue did not affect all collections or customers equally, those with recent content updates were most likely to notice discrepancies within their admin interface.** ‌ ‌ **Root cause Analysis** **The incident was caused by a recovery volume that had been used in a prior restore task but was not properly detached after use.When the affected server restarted as part of a scheduled release, it mounted this older volume instead of the correct one due to a conflict between system identifiers.** **This resulted in the search system presenting data that was current as of 30 June 2025, rather than the most recent state.** **Although the issue was initially reported by developers working on non-live content, it quickly escalated as production data inconsistencies were identified.** ‌ ### **Resolution Actions** **Once the problem was confirmed, Squiz engineers initiated a structured response to diagnose and resolve the issue.** **Key actions included:** * **Declaring a technical incident and halting other ongoing deployments to focus on resolution** * **Identifying that an outdated recovery volume was still attached** * **Shutting down the affected server and detaching the incorrect volume** * **Restarting the server, allowing it to mount the correct, current data volume** * **Verifying that the latest data was restored and consistent with expectations** * **Reviewing push API access logs to assist customers in re-sending any missing updates** **Recommendations and Follow-up Actions** **Enforce Cleanup After Restore Operations** Ensure that any recovery volumes or temporary resources used during troubleshooting are fully removed when no longer needed.**Improve Volume Identification** Change the way data volumes are mounted by using different device IDs that are unique and immutable, reducing the risk of volume conflicts after a reboot. ### **Review and Adjust Backup Naming Conventions** Eliminate automated naming systems that obscure the purpose of EBS volumes, so restore drives can be more easily identified.

Read the full incident report →

Started Jun 23, 2025, 01:34 AM UTC· resolved Jun 23, 2025, 03:31 AM UTC

1. investigating Jun 23, 2025, 01:34 AM UTC

   We are currently investigating a performance issue affecting DXP customers in Australia. We have identified the issues and are working to resolve it.

2. investigating Jun 23, 2025, 01:55 AM UTC

   We are continuing to investigate this issue with the DXP hosting environment and will post more updates once we have any more information on this issue.

3. identified Jun 23, 2025, 02:01 AM UTC

   We've deployed a change to resolve the the immediate issue and are monitoring while working on additional fixes.

4. monitoring Jun 23, 2025, 03:00 AM UTC

   We are continuing to monitor the fix in place to see what errors are solved and if any further issues appear. We shall post further once we have more information.

5. resolved Jun 23, 2025, 03:31 AM UTC

   We have now resolved the issue here and performance should be back to normal. Feel free to get in touch via My.Squiz.Net to let us know if you have any issues you believe are related.

6. postmortem Jun 26, 2025, 11:36 PM UTC

   # **Summary** **On Monday, June 23, 2025, customers may have experienced intermittent periods of degraded performance. This disruption was attributed to the exhaustion of specific IP address ranges within our private Amazon Web Services \(AWS\) networks. This was promptly detected, and our engineering teams swiftly initiated remediation efforts. The issue was identified and resolved within a short timeframe, restoring full service functionality and optimal performance levels.** #### **Customer Impact:** * **Incident Duration: 23 Jun 2025, 11:06- 23 Jun 11:33 \(AEST\)** * **Impact: We have noticed that only a small number of clients have been affected by this incident.** ## **Issue, Resolution, and Mitigation** **Root cause Analysis** **The underlying cause of this incident was the unanticipated depletion of available IP addresses within designated private subnets in our AWS environment. A confluence of factors, including a surge in specific resource utilization and the rapid provisioning of new instances, led to a temporary bottleneck in IP availability.** ‌ **Resolution Actions** ### **1. Identification:** * **Squiz Support Teams proactively identified errors relating to this issue** ### **2. Action:** * **Releasing recently provisioned IP addresses** ### **Follow-up Actions** ### **1. Squiz Actions:** * **To prevent future occurrences, Squiz has completed the migration of specific infrastructure to a larger subnet.**

Read the full incident report →

Started Jun 12, 2025, 06:29 PM UTC· resolved Jun 13, 2025, 03:01 AM UTC

1. investigating Jun 12, 2025, 06:29 PM UTC

   We are experiencing service degradation on the DXP environment. Currently investigating

2. investigating Jun 12, 2025, 06:29 PM UTC

   We are continuing to investigate this issue.

3. identified Jun 12, 2025, 06:39 PM UTC

   We have identified issues with Cloudflare which are affecting DXP

4. identified Jun 12, 2025, 07:08 PM UTC

   We are continuing to monitor the situation with our services which have been affected by an outage at Cloudflare. We will provide additional updates as they become available from the Cloudflare team.

5. identified Jun 12, 2025, 07:10 PM UTC

   We are continuing to monitor the situation with our services which have been affected by an outage at Cloudflare. We will provide additional updates as they become available from the Cloudflare team.

6. identified Jun 12, 2025, 08:19 PM UTC

   We are beginning to see partial recovery to some affected systems but not all yet as of this time. Squiz teams are continuing to monitor the situation at Cloudflare and test affected sites.

7. identified Jun 12, 2025, 09:15 PM UTC

   We have seen further improvements to site statuses around all affected platforms and our teams are continuing to track down any remaining issues to ensure full resolution before we switch into a fully Monitoring status. Further updates as they become available.

8. monitoring Jun 12, 2025, 09:22 PM UTC

   At this time services have been restored across the platform and Cloudflare has confirmed the same from their end. Squiz teams will continue to monitor the situation for any potential follow up issues and provide updates here as needed.

9. monitoring Jun 12, 2025, 11:03 PM UTC

   We are continuing to monitor all sites involved in the incident to look for any lingering effects of the issue and ensuring that they are thoroughly investigated by our Engineers and fixed. We shall post further updates here if there are any changes.

10. monitoring Jun 12, 2025, 11:43 PM UTC

    Our Engineers are continuing to Fix lingering issues caused by this incident and investigating to find any further issues. More information will be posted as we progress.

11. monitoring Jun 13, 2025, 12:13 AM UTC

    We are making progress towards fixing the lingering issues from the incident at hand and our engineers are continuing to test and ensure the performance of sites. We shall update again with any progress.

12. monitoring Jun 13, 2025, 01:16 AM UTC

    We are continuing to Monitor for ongoing issues caused by this outage, Our engineers have been able to resolve most of what we have found and are continuing to monitor for any new issues. We shall post a further update once we have more information.

13. monitoring Jun 13, 2025, 02:03 AM UTC

    Our Engineers have made progress towards fixing more of the lingering issue. They are also continuing to monitor for new issue, we shall post here once there are any more updates.

14. resolved Jun 13, 2025, 03:01 AM UTC

    We have seen the lingering issues now start to resolve now and Squiz is considering this incident to be resolved. Please get in touch with us at My.Squiz.Net if you see any further issues so that we can investigate separately.

15. postmortem Jun 16, 2025, 04:22 PM UTC

    ### Summary On June 12, 2025, Cloudflare experienced a significant service outage that impacted a wide array of Cloudflare services. This list of services included Workers KV, which Squiz utilizes for serving frontend content and backend administration UIs. Squiz’s Support teams were alerted to this issue following alerts received through our internal monitoring. Some customers may have experienced 503 errors throughout the incident. ### ‌‌Customer Impact Customer Impact: * Incident Duration: 12 Jun 2025, 18:10 - 13 Jun 03:01 \(UTC\) * Impact: Some customers experienced 503 errors when viewing pages and administration interfaces. * Customer systems/site pages were inaccessible during some/all of the impact window. * Cloudflare marked the issue as resolved at 20:28 UTC. However, some services were not fully resolved immediately; this included Workers KV. This was resolved at 03:01 UTC Cloudflare's status page: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp ### Root Cause The cause of this outage was due to a failure in the underlying storage infrastructure used by Cloudflare's Workers KV service, which is a critical dependency for many of Squiz's Cloudflare configuration and DXP services. Cloudflare have stated that part of this infrastructure is backed by a third-party cloud provider, which experienced an outage on June 12th and directly impacted availability of the KV service. ### Resolution Actions * Identification: Squiz Support Teams received a number of alerts indicating service disruption, leading to quick identification of the underlying cause. * Cloudflare Actions: Detailed in the Cloudflare Post Mortem; [https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/](https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/) * Squiz Actions: Squiz Support worked closely with Cloudflare, taking swift manual action to clear KV worker errors and restore services while full remediation was underway. ‌‌**Follow-up Actions** Cloudflare Actions: Squiz have engaged with Cloudflare to improve resiliency of the services that depend on Workers KV and their storage infrastructure. Cloudflare are already undertaking the actions detailed below: * \(Actively in-flight\): Bringing forward our work to improve the redundancy within Workers KV’s storage infrastructure, removing the dependency on any single provider. During the incident window we began work to cut over and backfill critical KV namespaces to our own infrastructure, in the event the incident continued. * \(Actively in-flight\): Short-term blast radius remediations for individual products that were impacted by this incident so that each product becomes resilient to any loss of service caused by any single point of failure, including third party dependencies. * \(Actively in-flight\): Implementing tooling that allows us to progressively re-enable namespaces during storage infrastructure incidents. This will allow us to ensure that key dependencies, including Access and WARP, are able to come up without risking a denial-of-service against our own infrastructure as caches are repopulated. Squiz Actions: * \(In progress\): Change the Cloudflare layer of the DXP to remove dependencies on Cloudflare Workers KV where possible, and provide safe fallbacks when the service is unavailable.

Read the full incident report →

Started Jun 05, 2025, 07:37 AM UTC· resolved Jun 05, 2025, 07:59 AM UTC

1. investigating Jun 05, 2025, 07:37 AM UTC

   Our monitoring has detected a performance degradation affecting UK Funnelback customers only. We are currently investigating the issue and will provide updates as soon as possible.

2. investigating Jun 05, 2025, 07:43 AM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 05, 2025, 07:50 AM UTC

   Our engineering teams have taken action to restore the service and we are now seeing Funnelback performance returning to normal levels. We will continue to monitor at this time.

4. resolved Jun 05, 2025, 07:59 AM UTC

   Search performance has returned to normal, and the incident is now resolved. We will continue to monitor closely, but no further impact is expected. Thank you for your patience.

5. postmortem Jun 06, 2025, 08:11 AM UTC

   ### Summary On June 5, 2025, Squiz experienced a service degradation affecting UK Funnelback DXP customers. A spike in automated bot traffic led to a strain on system resources, which slowed down search performance and caused occasional service issues for some customers. A major incident was declared at 07:29 UTC. This led to some traffic being temporarily rerouted to alternate endpoints. Mitigation efforts were put in place, including enhanced traffic filtering and bot management measures. Service began to recover at 07:40 UTC following the introduction of targeted blocks and Cloudflare mitigation. The incident was declared resolved after a 10-minute monitoring period, with full service confirmed stable by 07:56 UTC. ### ‌‌Customer Impact Customers in the UK using Funnelback DXP services experienced degraded performance, including slower search responses and intermittent 504 gateway timeout errors. The impact was isolated to the UK region. No disruption was reported in other regions. ### ‌Issue, Resolution, and Mitigation ### Root Cause The incident was caused by a sharp increase in automated traffic, specifically from bot activity targeting UK customers' search services. This spike in non-human traffic led to congestion in the system, triggering rate-limiting protections which also impacted legitimate traffic. ### Resolution Timeline * 07:29 UTC – Major incident declared and incident channel initiated. * 07:32 UTC – A large volume of automated traffic, primarily from a known search engine crawler was observed targeting UK customers’ endpoints.This surge in traffic caused excessive load on the system, triggering rate-limiting protections and impacting legitimate user traffic. * 07:36 UTC – Mitigation initiated & Cloudflare traffic filtering enabled. * 07:39 UTC – Click-ops rule blocks over 3,000 suspicious requests. * 07:40 UTC – Service begins to recover. * 07:56 UTC – Incident declared resolved following successful monitoring. ### Mitigation Actions * Bot traffic was identified and blocked using custom rules. * Rate limits were reviewed and applied selectively. * Cloudflare-based mitigation was enabled to provide immediate relief. * Enhanced monitoring was put in place to verify system recovery.

Read the full incident report →