Wyng experienced a minor incident on April 16, 2018, lasting 6h 5m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Apr 16, 2018, 12:03 PM UTC
Drag and drop campaigns with a Form, Photo or Video Upload component is not loading. We have identified the root cause and working on a fix.
- monitoring Apr 16, 2018, 12:51 PM UTC
The fix to the root issue was released, previously unavailable services are restored.
- resolved Apr 16, 2018, 06:09 PM UTC
Referring to the previous outage this morning. The root cause was `pip` — the python package manager used by the entire python community, released a new version (10.0.0) that includes a serious breaking change. The submission-service servers automatically upgraded (because we configured it to use latest) earlier this morning, and stopped working afterwards. The fix was locking down the pip version to last stable version, instead of using the latest. For details: https://github.com/pypa/pip/issues/5247 https://github.com/pypa/pip/issues/4805 We are also working on some improvements to prevent this type of issue affecting us again in the future.