Amplience incident
Virtual Staging Environment (VSE) Performance Issues
Amplience experienced a major incident on September 18, 2023 affecting Dynamic Content, lasting 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 18, 2023, 06:45 PM UTC
We have identified a failure with VSE. Our Operations and Engineering teams are currently investigating the cause and looking to fix the problem
- monitoring Sep 18, 2023, 06:52 PM UTC
Our Engineering team have rolled back a change which you should now see vse working as expected
- resolved Sep 18, 2023, 07:14 PM UTC
We have monitored traffic and error logs. We can confirm VSE is operational. We do apologise for any inconveniences this may have caused.
- postmortem Sep 19, 2023, 03:08 PM UTC
# **Summary** On Monday, September 18th, we experienced 500 errors from images using the Amplience Virtual Staging Environment endpoint. # **Root Cause** We discovered failures after a renewed certificate used on multiple services failed. One of the our internal services did not have an up to date root level certificate and was therefore unable to validate against the latest certificate change. This in turn caused the service to fail. Once we detected failures we begun the process of rolling back to the previous certificate and we are now in the process of working towards a permanent fix. # **Corrective and Preventive Measures** Internally, we are working towards a permanent fix going forward.