Amplience incident

Intermittent Stuck Scheduling and Publishing in Dynamic Content

Minor Resolved View vendor source →

Amplience experienced a minor incident on May 1, 2023 affecting Dynamic Content, lasting 2h 2m. The incident has been resolved; the full update timeline is below.

Started
May 01, 2023, 06:43 PM UTC
Resolved
May 01, 2023, 08:45 PM UTC
Duration
2h 2m
Detected by Pingoru
May 01, 2023, 06:43 PM UTC

Affected components

Dynamic Content

Update timeline

  1. investigating May 01, 2023, 06:43 PM UTC

    We are currently investigating this issue.

  2. identified May 01, 2023, 08:25 PM UTC

    The issue has been identified and a fix is being implemented.

  3. resolved May 01, 2023, 08:45 PM UTC

    The incident has been resolved and a post mortem report will be released outlining the issue and resolution.

  4. postmortem May 03, 2023, 03:30 PM UTC

    **Issue Summary** On the afternoon of Monday 1st May, we identified 3 issues within Dynamic Content whereby users were unable to see content items publish, receive webhooks and Schedule editions. **Root Causes** Our Operations team discovered a faulty shard after extensive testing was conducted in our QA environment. The faulty shard was replaced and we have monitored a healthy environment since this change. **Corrective Actions** Internally, we have discussed this at length to ensure integrity of service. Further monitoring is to be implemented to catch the errors earlier in our incident pipeline. We have also reviewed our processes to ensure a more efficient recovery of service. ‌If you would like to discuss this further, please contact your customer success manager \([[email protected]](mailto:[email protected])\)