implio incident
Custom machine learning predictions ignored under certain conditions
implio experienced a major incident on October 4, 2022, lasting 20h 44m. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 05, 2022, 04:31 PM UTC
A regression occurred between October 4, 13:52 CET and October 5, 10:36 CET, and caused custom machine learning model predictions to be ignored for some Implio teams, and under certain conditions. When those conditions were met, some moderation decisions during that timeframe were incorrect. - Affected *teams* were those with only one custom machine learning model (teams with multiple custom machine models for multiple languages were *not* affected). - Affected *items* were those with a language prediction (those without a language prediction were *not* affected). - Affected *rules* were those referring to the custom machine learning model prediction, exposed via the `$ml.status` variable (other rules were *not* affected). This regression was a side-effect of an improvement made to the platform. It escaped our tests and monitoring. After patching the regression, we've added the missing monitoring rules, so a similar incident would be picked up by our monitoring moving forward. We apologize for the impact this may have had on your moderation results. Please reach out to your Customer Success Manager for any question you may have.