Amplience incident

Dynamic Imaging Performance Issues

Major Resolved View vendor source →

Amplience experienced a major incident on May 29, 2023 affecting Dynamic Media, lasting 19m. The incident has been resolved; the full update timeline is below.

Started
May 29, 2023, 04:28 PM UTC
Resolved
May 29, 2023, 04:48 PM UTC
Duration
19m
Detected by Pingoru
May 29, 2023, 04:28 PM UTC

Affected components

Dynamic Media

Update timeline

  1. investigating May 29, 2023, 04:28 PM UTC

    We are aware of issues with the performance of image delivery and processing i.e. (i1.adis.ws). Our Engineering team are investigating and we will update you shortly with any further progress.

  2. resolved May 29, 2023, 04:48 PM UTC

    This incident has been resolved.

  3. postmortem May 30, 2023, 04:53 PM UTC

    **Issue Summary** On Monday 29th May we experienced a failed deployment of a TLS certificate which impacted a number of CNAMEs. Impacted customers would have seen requests to Cloudflare using [i8.amplience.net](http://i8.amplience.net/) and any custom CNAME configured on the certificate fail during the incident ‌ **Root Causes** We carried out a standard change \(pre-authorized and low-risk\) which failed before being deployed onto one of our CDN providers, Cloudflare. The certificate was re-issued and reapplied to all relevant CNAMES on our infrastructure to restore service. ‌ **Corrective Actions** Internally, we have discussed this at length to ensure the integrity of service. We have added in additional approval steps to our processes Further monitoring is to be implemented to monitor every certificate. We have also reviewed our processes to ensure a more efficient recovery of service. ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)