Fasterize Outage History

Started May 22, 2026, 09:33 AM UTC· resolved May 22, 2026, 03:18 PM UTC

1. investigating May 22, 2026, 09:33 AM UTC

   We currently have some issues on our European infrastructure. Impact on acceleration. Some pages can have some slowdowns. We're investigating but seems to be due to our hosting provider partial outage https://status.scaleway.com/incidents/qkj49g6g3ykp

2. monitoring May 22, 2026, 09:44 AM UTC

   Everything is back to normal but we're still monitoring. Root cause seems to be a power issue in a datacenter.

3. monitoring May 22, 2026, 09:53 AM UTC

   We are continuing to monitor for any further issues.

4. resolved May 22, 2026, 03:18 PM UTC

   Incident is closed since 11:30am but we keep on monitoring as the incident on our hosting provider is not fully resolved.

Read the full incident report →

Started May 20, 2026, 10:33 AM UTC· resolved May 21, 2026, 05:38 AM UTC

1. identified May 20, 2026, 10:33 AM UTC

   We are currently investigating an issue affecting website status checks in the console. Some websites may incorrectly appear with the status “Ping blocked” or “Sonde bloquée”. This does not necessarily mean that your website is unavailable. We will share an update as soon as possible.

2. resolved May 21, 2026, 05:38 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 20, 2026, 08:00 AM UTC· resolved May 20, 2026, 08:08 AM UTC

1. identified May 20, 2026, 08:00 AM UTC

   We are currently fixing an issue affecting flush operations through the API and dashboard. Our team is working to restore normal behaviour as quickly as possible. We will share an update once the issue is resolved.

2. resolved May 20, 2026, 08:08 AM UTC

   The issue affecting flush operations through the API and dashboard has been fixed. Flush actions are now working normally again.

Read the full incident report →

Started Feb 05, 2026, 12:00 PM UTC· resolved Feb 05, 2026, 12:00 PM UTC

1. resolved Feb 09, 2026, 05:46 PM UTC

   We have identified an issue affecting the accuracy of INP data collected between Thursday, 5 February and Monday, 9 February. During this period, INP metrics were incorrectly reported for Safari on iPhone when the navigation type was back-forward cache (BFCache). This behaviour is related to a known issue in WebKit, which causes inflated INP values in this specific scenario. More details are available in the related Safari bug report: https://bugs.webkit.org/show_bug.cgi?id=305251 The issue has been identified and mitigated on our side. Only the INP metric is impacted; other performance metrics remain unaffected. We apologise for any confusion this may have caused and recommend caution when analysing INP data for Safari iPhone traffic over this time window. If you have any questions, our team remains available.

Read the full incident report →

Started Nov 24, 2025, 02:50 PM UTC· resolved Nov 24, 2025, 03:43 PM UTC

1. investigating Nov 24, 2025, 02:50 PM UTC

   The Fasterzie Console is currently slow. We’re investigating.

2. identified Nov 24, 2025, 03:22 PM UTC

   The issue has been identified, our team is working on resolving it

3. resolved Nov 24, 2025, 03:43 PM UTC

   The incident has been resolved, everything is now operating normally

Read the full incident report →

Started Oct 09, 2025, 09:01 AM UTC· resolved Oct 09, 2025, 06:23 PM UTC

1. investigating Oct 09, 2025, 09:01 AM UTC

   Our platform is experiencing an outage, where possible traffic is automatically routed to origin to mitigate the incident. We identified the issue and are working on a fix to restore availability

2. identified Oct 09, 2025, 09:54 AM UTC

   We are still investigating and trying to restore the failing component.

3. identified Oct 09, 2025, 10:27 AM UTC

   The failing component is gradually recovering, and we are monitoring it until full restoration

4. monitoring Oct 09, 2025, 10:34 AM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Oct 09, 2025, 01:20 PM UTC

   Configuration updates are not possible from the Fasterize console. We are currently investigating.

6. investigating Oct 09, 2025, 03:35 PM UTC

   Acceleration is now fully operational. Configuration updates are still unavailable, and we are investigating the issue. Cache purges are also currently unavailable.

7. identified Oct 09, 2025, 05:04 PM UTC

   The issue has been identified and a workaround is being implemented. In the meantime, configuration updates and cache purges are unfortunately still unavailable. The workaround will unfortunately degrade the API performance but it should be temporary until we rollout a definitive solution. As soon as the fix is delivered, we'll monitor it to be sure that everything's ok. The postmortem will follow, as we also need to gather further elements from our provider. We deeply apologize for this incident and we're actively working to solve this issue as soon as possible.

8. resolved Oct 09, 2025, 06:23 PM UTC

   This incident has been resolved. The API and the dashboard are now able to handle correctly configuration updates and cache purges. As mentioned earlier, a postmortem will be written and available here.

9. postmortem Oct 13, 2025, 12:17 PM UTC

   An in-memory database cluster failure occurred leading to service unavailability across multiple Fasterize components — primarily the **Optimisation Engine** and the **API**. At **09:48**, an in-memory database cluster failure happens after restarting multiple nodes to release a new engine version. The database cluster began an automatic failover sequence, but each time a new node was promoted as primary, it **crashed under excessive connection load**. This cluster serves as a **cache layer** providing access to configurations. During the outage, Engine instances attempted to reconnect at a very high frequency and fell back to retrieving data directly from the main database.This fallback mechanism worked as intended until **10:32**, allowing our optimization engine to continue operating in **degraded mode**. At **10:32**, however, the **proxy layer** of our optimisation engine became **saturated in network resources**, rendering it unavailable from the front layer. When the proxy layer is unreachable, the platform automatically **unplugs the CDN and sites are served directly from their origin servers**, without Fasterize optimizations. Working with our hosting provider, we **reduced the Optimisation Engine cluster size at 11:45** to limit reconnection attempts to Redis. By **12:16**, the Redis cluster had stabilised and full service was restored. Later, at **14:15**, we detected that the **API was unable to write to the Redis cluster**. The root cause was a **security patch applied by the hosting provider**, which **restricted the use of some commands** in the cluster.The API was patched to remove usage of these commands, and full functionality was restored by **20:15**. ‌ ## **Impact** * **Duration:** 10h32 – 12:16 \(outage\), API issue until 20:15 * **Affected components:** Optimisation Engine, API * **User impact:** Most websites temporarily served unoptimised content directly from origin; Some websites experienced unavailability due to an dysfunctional DNS fallback mechanismAPI write operations failed, blocking configuration update. ## **Resolution Timeline** ‌ ## Action plan Short term: * Improve **alerting and visibility** on in-memory database cluster health and failover events. * Review the in-memory database connection logic of the engine to avoid too many reconnection attempts in case of disconnections and avoid a case preventing the start of the process in case of an in-memory database outage. * Adjust failover DNS logic to avoid redirection to the origin when the CDN/fronts are still able to accept the traffic. * Upscale in-memory database cluster in order to accept more connections Medium term: * Review and test **disaster recovery procedures** for in-memory database cache clusters, including the ability to quickly activate a passive cluster. Long term: * Re-architecture the engine to reduce the number of connections on the itn-memory database cluster

Read the full incident report →

Started Sep 25, 2025, 08:00 AM UTC· resolved Sep 25, 2025, 08:00 AM UTC

1. resolved Sep 25, 2025, 09:07 AM UTC

   A network maintenance caused a regression in the service responsible for extracting the True-Client-IP header from the X-Forwarded-For header. Around 6% of the platform trafic has been impacted. The issue has now been resolved. We apologize for the inconvenience.

Read the full incident report →

Started Sep 09, 2025, 01:11 PM UTC· resolved Sep 09, 2025, 01:51 PM UTC

1. monitoring Sep 09, 2025, 01:11 PM UTC

   Between 14:49 and 14:52 (Paris time), our services were briefly disrupted due to a Distributed Denial of Service (DDoS) attack. The issue has been fully resolved, and systems are operating normally. We appreciate your patience and understanding. Sorry for the inconvenience.

2. resolved Sep 09, 2025, 01:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 09, 2025, 10:13 AM UTC· resolved Sep 09, 2025, 10:13 AM UTC

1. resolved Sep 09, 2025, 10:13 AM UTC

   Between 11:35 a.m. and 11:55 a.m. (Paris time), the platform experienced performance slowdowns following a scheduled update impacting the load balancers.

2. postmortem Sep 10, 2025, 01:24 PM UTC

   # **🛠️ Incident Description** **On Tuesday, September 9th, 2025, Fasterize carried out a scheduled maintenance to modernize part of its private network infrastructure.** **During this operation, an unexpected configuration issue temporarily disrupted communication between some components of the platform. This resulted in an increased error rate on part of the traffic between 11:43 and 11:50 \(Paris time\).** # **📅 Timeline** * **11:40 – Update started on the first batch of machines** * **11:43 – Anomaly detected in communication between proxies and fronts** * **11:50 – Issue mitigated, service returned to normal** # **✅ Immediate Actions** * **Affected machines were taken out of rotation** * **Healthy machines were reactivated to absorb the traffic** * **Service stabilized within minutes of the anomaly** # **🚀 Next Steps** * **Strengthening our validation procedures during network migrations** * **Adding automated connectivity tests to detect similar issues earlier** * **Improving documentation and standardization of network configurations to minimize future risks**

Read the full incident report →

Started Sep 09, 2025, 12:30 AM UTC· resolved Sep 09, 2025, 12:30 AM UTC

1. resolved Sep 10, 2025, 01:28 PM UTC

   On September 9, Fasterize performed a planned network maintenance to replace an obsolete private network. As part of this operation, a new private IP address was introduced for our load balancers while keeping the existing ones in place. Later that afternoon, a DDoS attack led to the saturation and restart of one of the load balancers. Following this restart, the load balancer started using the new private IP address to communicate with the backend servers. Since this IP had not yet been registered as a trusted proxy, some requests were processed incorrectly.

2. postmortem Sep 10, 2025, 01:28 PM UTC

   ## **🛠️ Incident Report** **Date:** Tuesday, September 9, 2025 **Duration:** 14:50 – 18:30 \(Paris time\) **Impact:** ~23% of traffic ### **📌 Summary** On September 9, Fasterize performed a planned network maintenance to replace an obsolete private network. As part of this operation, a new private IP address was introduced for our load balancers while keeping the existing ones in place. Later that afternoon, a DDoS attack led to the saturation and restart of one of the load balancers. Following this restart, the load balancer started using the new private IP address to communicate with the backend servers. Since this IP had not yet been registered as a trusted proxy, some requests were processed incorrectly. ### **📈 Impact** * ~23% of traffic was affected between **14:50 and 18:30**. * Some users encountered **403 errors**. * In certain cases, client IP addresses were not correctly identified in the headers. ### **📅 Timeline** * **10:25** – New private IP address added to the load balancers. * **14:49** – DDoS attack causes one load balancer to restart. * **14:50** – First incorrect headers observed \(undetected at this stage\). * **16:02** – Support receives a ticket mentioning 403 errors. * **17:16** – Ticket escalated to the platform team. * **18:16** – Root cause identified and linked to the network update. * **18:30** – New IP address added to the trusted list; incident resolved. **Total duration of impact:** 3h40 ### **🔍 Root Cause** The new private IP address introduced during the maintenance was not yet included in the trusted proxy configuration. When one load balancer restarted, it began using this IP, leading to incorrect handling of client headers. ### **🚀 Next Steps** * All load balancer IPs have now been added to the trusted configuration across environments. * Documentation has been updated to include this step in future network changes. * Additional monitoring will be set up to detect unexpected private IPs in client headers. * Escalation procedures between support and platform teams will be reviewed to ensure faster response times.

Read the full incident report →

Started Jun 18, 2025, 07:41 AM UTC· resolved Jun 18, 2025, 08:55 AM UTC

1. investigating Jun 18, 2025, 07:41 AM UTC

   We currently have some issues on one of our european DC. Being fixed. Trafic is interrupted for a large portion of the customers. Really sorry for the inconvenience.

2. investigating Jun 18, 2025, 07:49 AM UTC

   The failover mechanism didn't trigger. We trigger it manually.

3. resolved Jun 18, 2025, 08:55 AM UTC

   This incident is now resolved. Post-mortem will follow but to sum up: the root cause is a change in a DNS record. During that change, the record pointing to our DC took a temporary wrong value that was captured by some edge servers and stored for one hour. This affected only a subset of edge servers and only a subset of the health checkers responsible for triggering the failover mechanism. This explains why the failover mechanism wasn't fully triggered.

4. postmortem Jun 19, 2025, 08:38 AM UTC

   **🧾 Incident Summary** As part of preparations for the summer sales and in response to a load handling issue observed on June 12, the tech team has planned an update of a DNS record pointing to the platform’s frontend layer. The goal was to review the way CDN Edge servers send traffic to the platform. Since the DNS record could not be edited in place, it was deleted and immediately recreated with a new routing policy. During the brief deletion \(~10 seconds\), some DNS resolvers fell back to a default wildcard record with a 3600s TTL, pointing to decommissioned IPs instead of returning NXDOMAIN. As a result, several CDN Edge servers cached these incorrect IPs and continued to use them for up to an hour, causing a global service outage on the affected POPs. Some health check probes also resolved to the wrong IPs but not enough of them failed at once to trip the health check thresholds \(which require consecutive failures to switch to a degraded state\). ‌ ### **📅 Timeline** ### **🧩 Root Cause Analysis** This outage was not due to an infrastructure failure but to a human error during maintenance without sufficient safeguards. We consider this as a major governance failure in infrastructure change management. The affected DNS zone was created over 12 years ago and currently holds legacy records. Over time, several have become outdated or unused. However, due to the risk of accidental deletion and unexpected impact, no regular cleanup has been performed. This lack of maintenance allowed a misconfigured wildcard record to persist, which was unintentionally triggered during the temporary deletion of a critical DNS record. We’ve now initiated a full audit of this DNS zone to identify, document, and progressively remove obsolete records. A cross-validation policy will be enforced before any future changes. Although a staging environment exists and is used to validate infrastructure changes, the specific scenario here — related to active CDN traffic and DNS behavior during the few seconds of deletion — was not anticipated. Due to the distributed and asynchronous nature of CDN DNS propagation and caching \(per POP\), the issue could hardly be replicated in a staging environment. ### **✅ Immediate Fixes** * Re-creation of the DNS record with the correct configuration * Manual checks to ensure resolvers and CDN POPs are now resolving to the correct origin ### **🔒 Preventive countermeasures** **Short-term:** * Freeze on infrastructure changes for two weeks **Medium-term:** * Improve the staging environment to better simulate CDN-specific issues * Clean up outdated records in the platform's DNS zone * Formalize an emergency fallback procedure * Introduce more logical zones to avoid widespread impact across clients ‌ ### **Conclusion** This incident highlights how even short-lived DNS misconfigurations can cause major disruptions in distributed systems like CDNs.Rigorous TTL management and better anticipation of critical DNS usages are essential to avoid similar outages in the future. Despite this incident, the technical and customer success team remains fully committed to delivering a smooth and successful sales period for our clients.A temporary change freeze is in place, and additional planning and capacity measures are being taken to ensure high availability and reliability throughout this peak traffic period.

Read the full incident report →

Started Jun 11, 2025, 01:41 PM UTC· resolved Jun 11, 2025, 04:50 PM UTC

1. investigating Jun 11, 2025, 01:41 PM UTC

   We currently have some issues on our european infrastructure. We're investigating. Speeding-up is disable. Traffic should be redirected to origins.

2. identified Jun 11, 2025, 02:02 PM UTC

   Issue has been identified. Some load balancers in one of our datacenter have crashed and have been restarted. Traffic was interrupted at 3:31PM and is back to normal since 3:42PM CET.

3. monitoring Jun 11, 2025, 03:02 PM UTC

   We're still monitoring.

4. resolved Jun 11, 2025, 04:50 PM UTC

   Incident is now closed. Sorry for the inconvenience. A post-mortem will follow.

5. postmortem Jun 12, 2025, 10:29 AM UTC

   ## **Summary** On June 12 2025, Fasterize experienced a service disruption for a subset of customers due to a large-scale DDoS attack. From 15:31 to 15:42 \(UTC\+2\) the surge in traffic overloaded the zone’s load balancers, resulting in a temporary fallback to origin servers for affected sites. Total outage time was 11 minutes, with full platform stability restored at 15:42. ## **Timeline \(UTC\+2\)** * **15:31 – 15:42**: DDoS attack peaks at ~8 million requests overall, spiking to 50 000 req/s. Load balancers in the zone restart repeatedly. * **15:34**: Internal alerts detect the saturation \(detection time: 3 min\). Health-check system routes traffic directly to origin servers. * **15:42:** Traffic normalises; services in the affected zones recover \(time to full resolution: 11 min\). ## **What Happened** The volumetric DDoS flooded the load balancers, driving CPU utilisation to 100 %. Although our automatic health-check mechanism successfully redirected traffic to customer origin servers \(for those customers' whose stack permits this automatic redirection\), the repeated load-balancer restarts caused an 11-minute service degradation for sites relying on that zone. We had increased load-balancer capacity after a similar event on May 15, but this proved insufficient for the size of yesterday’s attack, highlighting a design limitation in relying on provider-managed load balancers. ‌ ## Impact * **Severity level:** 1 — Unplanned downtime affecting multiple production websites * **Detection time:** 3 minutes * **Time to full recovery:** 11 minutes During the incident, visitors to impacted sites experienced 502 errors until they were served directly from origin infrastructure, potentially experiencing higher latency. ## **What We're Doing** ### Short-term improvements \(in progress\) * Re-architecture to avoid load balancer issues ### Long-term improvements * **Additional zones:** Bringing online new zones to spread traffic and isolate attacks more effectively. * **Rate-limit hardening:** Evaluating two third-party CDNs that offer native volumetric-attack rate limiting.

Read the full incident report →

Started May 28, 2025, 07:00 AM UTC· resolved May 28, 2025, 07:00 AM UTC

1. resolved May 28, 2025, 07:53 AM UTC

   What happened: This morning at 8:52, we made planned adjustments to our infrastructure in order to improve the reliability of the connection between our platform and CDN. During the transition, the service experienced a short disruption due to unexpected load balancer restart. Impact: The site was briefly unavailable for approximately one minute generating error 502. Resolution: The situation was quickly identified and resolved by 08:52 (GMT+2) . Everything is now back to normal and fully operational. We’re sorry for the inconvenience and are taking steps to prevent this from happening again.

Read the full incident report →

Started Feb 25, 2025, 10:10 AM UTC· resolved Feb 25, 2025, 05:06 PM UTC

1. identified Feb 25, 2025, 10:10 AM UTC

   Acceleration was disabled between 10:45 AM GMT+1 and 11 AM GMT+1 due to a misconfiguration of the global platform health checks. These health checks are currently being reviewed (but the platform is actually healthy).

2. identified Feb 25, 2025, 03:37 PM UTC

   Root cause has been identified. A post-mortem is being written. We're gonna set up more robust global platform health checks.

3. monitoring Feb 25, 2025, 04:37 PM UTC

   Fix has been deployed, everything is back to normal (the platform is properly monitored again). We're still monitoring.

4. resolved Feb 25, 2025, 05:06 PM UTC

   Incident is now closed. Sorry for the inconvenience. A post-mortem will follow.

5. postmortem Feb 25, 2025, 05:06 PM UTC

   # Healthcheck in error Date of post mortem : 25/02/2025 Participants of the post mortem : Anthony Barré # Description of the incident Following a change in the Fasterize API, **the health checks of the platform turned red**, indicating an unavailability of the platform while it was actually healthy. These health checks were configured to query a URL associated with **a domain name that was no longer present** in the configuration database. ### **Immediate corrective action** * **Temporary deactivation** of health checks. * **Rollback** of the API configuration change. # Facts & Timeline All times are UTC\+1. # Analysis ## **Technical context** * The health checks validate the availability of the platform through several layers \(proxy, load balancer, workers, etc …\) * The failure of the health checks was due to errors at the proxy level, a symptom of the absence of configuration associated with the domain used. * At the proxy level, the health check response is adapted by region to respond to the health checks associated with each region. * An outdated process, that was still running despite no longer being required, caused issues when it updated the configuration used by proxies with incorrect data. ## **Root cause analysis** The API configuration change caused an unstable state. Indeed, an API update did not take place immediately. Thus, some API pods were reading the health check configuration from one environment and others from another environment. This corrupted the database used by the proxies of our main environment. The proxies no longer responded with 200 but with error code to health check requests. ## **Impact of incident** 🏠 **Affected customers** : **All customers** were impacted by the lack of acceleration. 🔴 **Specific issue** : **Two customers** experienced major disruptions because their origins did not allow them to receive traffic directly from the Internet. 📊 **Incident metrics** * **Severity**: **1** \(service shutdown impacting a large number of users\). * ⏱ **Detection time**: **4 minutes**. * 🛠 **Resolution time**: **20 minutes**. # Action Plan ## Short term \(immediate - 1 week\) * Review the platform's health checks so that they are more robust * Disable and delete the obsolete monitoring process ✅ * Fix the deployment of configmaps in the helm charts. ## Medium term * Review the API configuration to clarify the use of region-related fields and avoid side effects.

Read the full incident report →

Started Jan 01, 2025, 09:00 PM UTC· resolved Jan 01, 2025, 09:00 PM UTC

1. resolved Jan 02, 2025, 10:44 AM UTC

   We encountered a minor issue affecting the purge of image metadata and CSS/JS bundles. While the purge process experienced some disruptions, this had no noticeable impact on website functionality. Pages and static files were purged correctly.

Read the full incident report →

Started Dec 11, 2024, 08:30 AM UTC· resolved Dec 11, 2024, 08:30 AM UTC

1. resolved Dec 11, 2024, 01:53 PM UTC

   Timeframe: 9:30 AM to 10:15 AM During this period, pages were served without optimization by the platform. There was no downtime, and traffic continued to be served seamlessly with original (non-optimized) pages. Cause: The issue was triggered by a recent release that unexpectedly introduced a high load on the platform. Resolution: We promptly identified the issue and rolled back the release to restore normal operations. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Read the full incident report →