Resolved after 35 minutes, affecting implio core, on November 30, 2021.
On 11/30/2021 at 17:22 CET, the processing of items got very slow, after an unattended upgrade in one of our internal services. This service added metadata that it wasn’t able to handle, causing instability in our platform. We tried to rollback and gracefully handle these items as fast as possible. At 17:50 CET, we came up with a hotfix that ignored this metadata, so the processing of items could continue. The hotfix had an unforeseen consequence that caused remote decisions submitted via Update requests to be ignored by our system. This may have caused erratic or duplicate moderation decisions from Implio. The definitive fix was delivered at 12/01/2021 15:07 CET. This problem should have been spotted by our continuous integration process. We’re taking this problem very seriously and putting additional tests in place to prevent future occurrences of similar problems. We’re sorry for the inconvenience this problem may have caused. Thank you for your understanding.
Affected components
Update timeline
- investigating Nov 30, 2021, 04:48 PM UTC
Processing of documents is currently blocked, we're investigating
- monitoring Nov 30, 2021, 04:53 PM UTC
We've idenfitied and provided a fix, now we're moniroting and trying to understand what happened
- monitoring Nov 30, 2021, 05:11 PM UTC
Almost all items in processing are being processed right now. We are really sorry for the delayed of processing time. We have some action points to try to avoid this in the future.
- monitoring Nov 30, 2021, 05:13 PM UTC
We are continuing to monitor for any further issues.
- resolved Nov 30, 2021, 05:23 PM UTC
The issue is now fully fixed. Again, we are really sorry for the delayed of processing time. We have some action points to try to avoid this in the future.
- postmortem Dec 02, 2021, 02:18 PM UTC
On 11/30/2021 at 17:22 CET, the processing of items got very slow, after an unattended upgrade in one of our internal services. This service added metadata that it wasn’t able to handle, causing instability in our platform. We tried to rollback and gracefully handle these items as fast as possible. At 17:50 CET, we came up with a hotfix that ignored this metadata, so the processing of items could continue. The hotfix had an unforeseen consequence that caused remote decisions submitted via Update requests to be ignored by our system. This may have caused erratic or duplicate moderation decisions from Implio. The definitive fix was delivered at 12/01/2021 15:07 CET. This problem should have been spotted by our continuous integration process. We’re taking this problem very seriously and putting additional tests in place to prevent future occurrences of similar problems. We’re sorry for the inconvenience this problem may have caused. Thank you for your understanding.
Your product runs on more than implio
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services