Amplience incident

Staging links return 504 errors

Major Resolved View vendor source →

Amplience experienced a major incident on March 4, 2025 affecting Content Hub and Dynamic Media, lasting 5m. The incident has been resolved; the full update timeline is below.

Started
Mar 04, 2025, 04:16 PM UTC
Resolved
Mar 04, 2025, 04:22 PM UTC
Duration
5m
Detected by Pingoru
Mar 04, 2025, 04:16 PM UTC

Affected components

Content HubDynamic Media

Update timeline

  1. investigating Mar 04, 2025, 04:16 PM UTC

    We are currently investigating an issue where staging (Virtual Staging Environements) links return 504 error codes. Existing published images are not impacted.

  2. monitoring Mar 04, 2025, 04:20 PM UTC

    A fix has been implemented, and we are monitoring the results.

  3. resolved Mar 04, 2025, 04:22 PM UTC

    This incident is now resolved.

  4. postmortem Mar 07, 2025, 02:00 PM UTC

    Incident Start Time: 15.56 GMT Incident End Time: 16.20 GMT ‌ **Issue summary** On Tuesday, 4th March, users of Virtual Staging Environments \(VSE\) experienced 504 Gateway Timeouts errors when attempting to preview content and assets. The delivery of published content and assets were not impacted. ‌ **Root cause** Earlier in the day, we deployed a low-risk update to the VSE stack, which was successfully validated post-deployment. However, the service was not scaled appropriately to accommodate the traffic demands, ultimately reaching capacity and triggering 504 errors. ‌ The issue was resolved by scaling the service appropriately. ‌ **Corrective Actions** Future releases to the VSE will include proactively monitoring the drift within Cloudformation to ensure service capacity meets traffic requirements. ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)