Amplience experienced a major incident on March 22, 2024 affecting Dynamic Media, lasting 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 22, 2024, 09:12 AM UTC
We are currently investigating reports of broken images
- monitoring Mar 22, 2024, 09:26 AM UTC
A third-party supplier experienced an outage which impacted some of our services. They have implemented a fix and we are monitoring our services. https://www.cloudflarestatus.com/incidents/x1232bd91lkp
- resolved Mar 22, 2024, 09:44 AM UTC
The issue is now resolved. We will conduct an internal review and post our findings in a postmortem
- postmortem Mar 25, 2024, 04:34 PM UTC
**Issue Summary:** During the morning of Friday, March 22nd, 0.1% of Dynamic Imaging resulted in 404 errors. Initially, we attributed this issue to a potential incident with Cloudflare. [https://www.cloudflarestatus.com/incidents/x1232bd91lkp](https://www.cloudflarestatus.com/incidents/x1232bd91lkp) Amplience boasts a multi-layer CDN infrastructure, ensuring continuous delivery. We were able to redirect our traffic away from Cloudflare within 15 minutes while we investigated further. Upon further analysis, we discovered that it stemmed from a recent change pushed out by Cloudflare to all environments. **Root Causes:** Cloudflare, a third-party supplier, implemented a modification to HTTP/2, which took effect in our environment by default at 00:00. Disabling this modification resolved the service disruption. **Corrective Actions:** Upon being informed by Cloudflare about the change, we proceeded with an internal risk assessment. This Incident has highlighted areas where further improvements are warranted.