Wyng incident
New content ingestion from Instagram and Twitter is not working
Wyng experienced a major incident on February 27, 2020 affecting Instagram Content Stream and Content Management, lasting 1d 11h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 26, 2020, 04:23 PM UTC
Since 10:35 AM US eastern time, new content ingestion from Instagram and Twitter is not working. Some content moderation features may also not be available. Content API queries are working normally. We are currently investigating this issue.
- identified Feb 26, 2020, 05:42 PM UTC
The issue has been identified and we are working on a fix.
- identified Feb 26, 2020, 08:01 PM UTC
A fix is in process, and will require some time to complete. In the interim, all existing content galleries will continue to display data previously collected or tagged. However the following capabilities will not work until the issue is fully resolved: 1. New content ingestion from Instagram, Twitter, or Direct Upload. 2. Adding tags to content. 3. Updating permission to use / rights management. Content moderation, and adding/updating conversion units is working normally.
- identified Feb 26, 2020, 08:49 PM UTC
We are continuing to work on a fix for this issue.
- identified Feb 26, 2020, 08:55 PM UTC
New Instagram and Twitter content is being collected and queued up. Once the fix is complete, the backlog of queued content will be processed as usual, and visible within your Content Management dashboard.
- identified Feb 27, 2020, 05:12 AM UTC
The resolution process is proceeded as expected, and will continue to take more time before completion.
- identified Feb 27, 2020, 08:04 PM UTC
The process to resolve is continuing normally, but will continue to take more time. We are unable to provide an ETA on resolution at this time, but we will post an update later today.
- identified Feb 28, 2020, 05:40 AM UTC
The time intensive part of the resolution process has completed, and processing of backlogged content and new content is recovering. All content moderation features are operating normally. We are reviewing status of all production systems and will provide a further update tomorrow at 7 AM U.S. eastern time.
- monitoring Feb 28, 2020, 05:43 AM UTC
Systems are recovering, and backlogged content is being processed. We are continuing to monitor.
- monitoring Feb 28, 2020, 12:02 PM UTC
All Instagram and Twitter content collected during the outage is in our backlog, and being processed. Other affected content features are working normally. We will continue to update on this issue until the backlog processing has completed.
- resolved Feb 28, 2020, 04:27 PM UTC
Collection of content from Instagram and Twitter, and all content features impacted by this incident, are operating normally. The full backlog of Twitter content has been processed. The backlog of Instagram content is still being processed, and will complete over the next few hours. [EDITED AT 2:30 PM EST: The full backlog of Instagram content has been processed.]