Amplience Outage History

**Incident Start Time:** 05.07 BST **Incident End Time:** 05.52 BST ‌ **Issue Summary:** An incident occurred with one of our back end databases. This caused degraded performance to DC-API and some Dynamic Content Editions were in a stuck status. ‌ **Root Cause:** Some inefficient queries to our back end database caused our primary node to fail over to our secondary node. ‌ **Corrective Actions:** We have already introduced corrective mitigations with additional indexes and working with MongoDB we have further improvements planned.

**Incident Start Time:** 19.49 BST **Incident End Time:** 22.03 BST **Issue Summary:** An incident occurred within Dynamic Content that required a rollback to a previous version. In the Development Tab of Dynamic Content, search indexes were returning zero records. **Root Cause:** The issue was introduced in a recent Dynamic Content release, which introduced a new Content Security Policy \(CSP\) header. This header unintentionally blocked Algolia search index requests, resulting in searches returning no records. **Corrective Actions:** * Improve validation and regression testing around CSP-related changes to ensure third-party services such as Algolia continue to function as expected. * Whitelist the security headers in a future release.

**Incident Start Time:** Friday, 8th May, 09.57 BST **Incident End Time:** Monday, 12th May, 14.18 BST ‌ **Issue Summary:** An incident occurred within Dynamic Content that required an immediate rollback to a previous version. Users were unable to navigate through different Repositories. ‌ **Root Cause:** The issue was introduced in the latest Content Management App release, which included a new multi-select feature for the repository library. The feature caused unintended behaviour that impacted repository navigation functionality. ‌ **Corrective Actions:** * Review release verification procedures to improve early detection of UI interaction issues.

**Incident Start Date** 19:40 BST **Incident End Date** 19:54 BST ‌ **Issue Summary** On Wednesday evening, users were unable to log into [app.amplience.net](http://app.amplience.net) for a period of 14 minutes. ‌ **Root Cause** Our third-party authentication provider, Auth0, experienced a DNS outage caused by a misconfiguration on their side. This incident also impacted some of our alerting capabilities, which we are currently reviewing with our partners. ‌ **Resolution** Auth0 resolved their DNS issue, restoring normal authentication services. **Corrective Actions** * Review and strengthen alerting to ensure visibility is maintained during third-party DNS-related incidents * Work with Auth0 to understand the full scope of the misconfiguration and their preventative measures

Incident Start Time: 15.56 GMT Incident End Time: 16.20 GMT ‌ **Issue summary** On Tuesday, 4th March, users of Virtual Staging Environments \(VSE\) experienced 504 Gateway Timeouts errors when attempting to preview content and assets. The delivery of published content and assets were not impacted. ‌ **Root cause** Earlier in the day, we deployed a low-risk update to the VSE stack, which was successfully validated post-deployment. However, the service was not scaled appropriately to accommodate the traffic demands, ultimately reaching capacity and triggering 504 errors. ‌ The issue was resolved by scaling the service appropriately. ‌ **Corrective Actions** Future releases to the VSE will include proactively monitoring the drift within Cloudformation to ensure service capacity meets traffic requirements. ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)

Incident Start Time: 19/01/2025 22.00 GMT Incident End Time: 21/01/2025 12.30 GMT ‌ **Issue summary** On Sunday night, some bulk ingestion jobs were initiated but did not complete, resulting in them becoming stuck. ‌ **Root cause** This was due to unexpected behaviour from routine repair services on our back-end database. ‌ **Corrective actions** Additional alarms have been implemented to monitor and capture this new behaviour. Upgrades to this part of our platform are already underway to improve stability and performance. ‌ ‌ ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)