Amplience experienced a major incident on July 25, 2023 affecting Content Hub, lasting 3h 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 25, 2023, 02:01 PM UTC
We are currently investigating a failed shard affecting assets loading on Content Hub
- investigating Jul 25, 2023, 02:01 PM UTC
We are currently investigating a failed shard affecting assets loading on Content Hub
- monitoring Jul 25, 2023, 02:19 PM UTC
We have identified the issue and a fix applied. We can see assets are becoming available again and we will continue to monitor the situation.
- identified Jul 25, 2023, 02:35 PM UTC
We have identified a second failure which will directly affect Assets loading in Content Hub. We will continue to work on a secondary fix.
- monitoring Jul 25, 2023, 02:41 PM UTC
We identified a second failure which will directly affect Assets loading in Content Hub. We will continue to monitor as shards recover.
- resolved Jul 25, 2023, 05:19 PM UTC
We have rolled out a change to mitigate the failed shards, we are now seeing stability across Content Hub and will continue to monitor this evening.
- postmortem Jul 31, 2023, 02:28 PM UTC
**Issue summary** On the afternoon of Tuesday 25th July, we identified an issue with loading assets in Content Hub. **Root Causes** We identified shard failures which contributed to the loss of asset visibility from the Content Hub Interface. This also affected uploading new assets and updating due to the loss of visibility into existing and new assets. **Corrective actions** Internally, we have conducted discussions on how to enhance our ability to mitigate such a failure from occurring again. New research and development will be conducted around existing systems as well as looking at new methods to avoid disruption to live and existing content within the platform.