Wyng incident
Some user generated content is not being displayed
Wyng experienced a minor incident on May 1, 2020 affecting Experiences and Content Management, lasting 19h 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 01, 2020, 03:07 PM UTC
Some user generated content is not being displayed in galleries, and in the content management dashboard. This is affecting multiple clients. We are investigating.
- investigating May 01, 2020, 03:35 PM UTC
We have verified all the missing content is in our database. The problem is in the API layer. We are continuing to investigate.
- identified May 01, 2020, 05:20 PM UTC
We are continuing to investigate, but have identified a potential cause in our Elastic Search cluster.
- identified May 01, 2020, 06:37 PM UTC
We have verified the issue is due to a replication problem between nodes of our Elastic Search cluster. We are working with Amazon Web Services tech support, who manages that system for us, to resolve the problem. In the interim, some clients may see missing content restored, but that may be intermittent, until the underlying issue is resolved.
- identified May 01, 2020, 08:56 PM UTC
We are continuing to work with Amazon Web Services tech support, who manages our ElasticSearch cluster, to resolve the problem.
- identified May 02, 2020, 01:20 AM UTC
We are continuing to work with Amazon on resolution of the issue. They have completed their analysis and have specific recommendations, which we are reviewing and will implement over the next 12-18 hours.
- monitoring May 02, 2020, 10:36 AM UTC
A fix has been implemented and we are monitoring results. Clients should begin to see all missing content in galleries, and within their content dashboard.
- resolved May 02, 2020, 11:03 AM UTC
The incident has been resolved. We have verified content is restored in a sample of client accounts that reported issues. If you have any further issues, please flag them to support.