Amplience Outage History

Started Mar 04, 2025, 04:16 PM UTC· resolved Mar 04, 2025, 04:22 PM UTC

1. investigating Mar 04, 2025, 04:16 PM UTC

   We are currently investigating an issue where staging (Virtual Staging Environements) links return 504 error codes. Existing published images are not impacted.

2. monitoring Mar 04, 2025, 04:20 PM UTC

   A fix has been implemented, and we are monitoring the results.

3. resolved Mar 04, 2025, 04:22 PM UTC

   This incident is now resolved.

4. postmortem Mar 07, 2025, 02:00 PM UTC

   Incident Start Time: 15.56 GMT Incident End Time: 16.20 GMT ‌ **Issue summary** On Tuesday, 4th March, users of Virtual Staging Environments \(VSE\) experienced 504 Gateway Timeouts errors when attempting to preview content and assets. The delivery of published content and assets were not impacted. ‌ **Root cause** Earlier in the day, we deployed a low-risk update to the VSE stack, which was successfully validated post-deployment. However, the service was not scaled appropriately to accommodate the traffic demands, ultimately reaching capacity and triggering 504 errors. ‌ The issue was resolved by scaling the service appropriately. ‌ **Corrective Actions** Future releases to the VSE will include proactively monitoring the drift within Cloudformation to ensure service capacity meets traffic requirements. ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)

Read the full incident report →

Started Feb 13, 2025, 06:04 PM UTC· resolved Feb 13, 2025, 08:17 PM UTC

1. investigating Feb 13, 2025, 06:04 PM UTC

   We are currently investigating reports of intermittent delivery of Dynamic Content.

2. investigating Feb 13, 2025, 06:59 PM UTC

   Our Operations team are still investigating the issue

3. identified Feb 13, 2025, 07:43 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 13, 2025, 08:10 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 13, 2025, 08:17 PM UTC

   This incident has been resolved.

6. postmortem Feb 20, 2025, 09:27 AM UTC

   **Update 03/03/2025** Incident Start Time: 17.15 GMT Incident End Time: 20.30 GMT ‌ **Issue summary** An incident occurred where some customers using C1 for Content Delivery experienced 403 errors on localized content. ‌ **Root cause** Despite extensive investigation, we were unable to determine a definitive root cause. Logs and historical data did not reveal a consistent triggering event or anomaly in platform behavior. ‌ We’ve taken some actions: * Implemented enhanced monitoring to proactively detect the issue and prevent customer disruption. * Created a run book outlining diagnostic steps for a quicker triage in case of reoccurance ‌ **Next Steps** * Continue monitoring for any recccurences * The incident has been escalated to our Engineering team, with a focus on improving platform reliability and performance. * Root Cause Analysis will be revisited if new evidence emerges. ‌ While the root cause remains unknown, the steps above will help mitigate future occurrences. ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\) ‌ ============================================== ‌ 20/02/2025 Update: The investigation into the root cause is still ongoing. We have identified some contributing factors, but we have not yet confirmed the root cause. We will provide an update once the investigation is complete and RCA is finalised. We appreciate your patience.

Read the full incident report →

Started Jan 21, 2025, 10:40 AM UTC· resolved Jan 21, 2025, 02:41 PM UTC

1. investigating Jan 21, 2025, 10:40 AM UTC

   We are currently investigating the cause of the bulk upload issues experienced over the last 24 hours, we will provide further updates as soon as possible.

2. identified Jan 21, 2025, 12:36 PM UTC

   Our operations team have confirmed the backlog of stuck processing jobs have now cleared.

3. monitoring Jan 21, 2025, 02:32 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 21, 2025, 02:41 PM UTC

   This incident has been resolved.

5. postmortem Jan 23, 2025, 11:11 AM UTC

   Incident Start Time: 19/01/2025 22.00 GMT Incident End Time: 21/01/2025 12.30 GMT ‌ **Issue summary** On Sunday night, some bulk ingestion jobs were initiated but did not complete, resulting in them becoming stuck. ‌ **Root cause** This was due to unexpected behaviour from routine repair services on our back-end database. ‌ **Corrective actions** Additional alarms have been implemented to monitor and capture this new behaviour. Upgrades to this part of our platform are already underway to improve stability and performance. ‌ ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)

Read the full incident report →

Started Jan 13, 2025, 06:42 PM UTC· resolved Jan 13, 2025, 07:58 PM UTC

1. investigating Jan 13, 2025, 06:42 PM UTC

   We are aware of an issue using enums in the content form screen, we are looking into this and will provide further updates soon.

2. identified Jan 13, 2025, 07:19 PM UTC

   The issue has been identified and a rollback is being implemented.

3. monitoring Jan 13, 2025, 07:51 PM UTC

   We have implemented a rollback plan and are currently confirming this has fixed the issue.

4. resolved Jan 13, 2025, 07:58 PM UTC

   This incident has been resolved.

5. postmortem Jan 16, 2025, 01:54 PM UTC

   Incident Start Time: 16.11 GMT Incident End Time: 19.55 GMT ‌ **Issue summary** An incident occurred within Dynamic Content that necessitated an immediate rollback to a previous version. The symptom of this incident was observed during content creation, where a schema using enums did not list any values. ‌ **Root cause** A recent Dynamic Content release introduced a critical bug. ‌ **Corrective actions** To restore service, we immediately rolled back to a previous version. ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)

Read the full incident report →

Started Dec 01, 2024, 07:32 PM UTC· resolved Dec 01, 2024, 11:05 PM UTC

1. investigating Dec 01, 2024, 07:32 PM UTC

   We are currently investigating an issue with VSEs which may experience degraded performance

2. investigating Dec 01, 2024, 07:56 PM UTC

   We have found and resolved the issue

3. resolved Dec 01, 2024, 11:05 PM UTC

   The incident was resolved at 19:56 and no further issues were seen.

4. postmortem Dec 04, 2024, 10:13 AM UTC

   **Incident Report** **Incident Start Date:** 01/12/2024 17:50 **Incident End Date:** 01/12/2024 19:56 **Summary** On the evening of December 1st, our monitoring systems detected degraded performance in our Virtual Staging Environment. During this time, some customers may have encountered 5XX errors when previewing images. It is important to note that this issue was isolated to image previews and **did not** impact the dynamic image service. **Root Cause** The disruption was triggered by an unexpected surge in traffic to our Virtual Staging Environment. **Resolution** To resolve the issue, we promptly scaled up the service to handle the increased load.

Read the full incident report →

Started Oct 30, 2024, 09:33 PM UTC· resolved Oct 31, 2024, 12:52 AM UTC

1. investigating Oct 30, 2024, 09:33 PM UTC

   We are currently looking into an issue with Video Transcoding. We will update shortly once we have investigated further.

2. identified Oct 30, 2024, 09:48 PM UTC

   We have identified the issue and looking to roll back to a working version. We will update you once we have confirmed the roll back was successful.

3. identified Oct 30, 2024, 10:43 PM UTC

   We are testing a fix that is in place, we will update shortly once we have confirmed the fix is working as expected.

4. identified Oct 30, 2024, 11:11 PM UTC

   Video Transcoding is partially resolved. All new transcodes will process as normal. We are working on a fix for any affected videos stuck during the issue.

5. resolved Oct 31, 2024, 12:52 AM UTC

   The issue has now been resolved.

6. postmortem Nov 01, 2024, 01:09 PM UTC

   **Incident Start Date:** 30/10/2024, 15:10 GMT **Incident End Date:** 30/10/2024, 23:30 GMT ‌ **Summary:** During this time, some users may have encountered a video processing status that appeared "In Progress" indefinitely. **Root Cause:** While videos were processed successfully, the system was unable to confirm this to end users due to an authentication error following a recent release. As a result, the processing status appeared stuck, despite completion in the backend. **Resolution:** To resolve this, we rolled back the affected release and restored full functionality. We were able to identify all videos impacted and reprocess the correct status code \(In progress to complete\). We’re also implementing additional safeguards to prevent similar issues in the future

Read the full incident report →

Started Oct 29, 2024, 01:13 PM UTC· resolved Oct 30, 2024, 01:19 PM UTC

1. investigating Oct 29, 2024, 01:13 PM UTC

   We are investigating an incident where customers using SFCC may experience issues with copied editions being scheduled into the wrong slot.

2. investigating Oct 29, 2024, 05:00 PM UTC

   We are still investigating this issue and we will share a further update shortly

3. identified Oct 29, 2024, 07:02 PM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Oct 30, 2024, 01:19 PM UTC

   This incident has been resolved and a fix has been released.

5. postmortem Nov 01, 2024, 01:08 PM UTC

   **Summary:** A small number of power users encountered issues when editing Editions, where slot data was occasionally saved incorrectly. ‌ **Root Cause:** Our investigation identified a race condition occurring when navigating between slots, which led to the issue. However, this was an edge case that did not affect all users or every Edition. ‌ **Resolution:** We deployed a hotfix to address this issue for future Editions and ensure stability across all users.

Read the full incident report →