implio incident

Custom machine learning predictions ignored under certain conditions

Major Resolved View vendor source →

implio experienced a major incident on October 4, 2022, lasting 20h 44m. The incident has been resolved; the full update timeline is below.

Started
Oct 04, 2022, 11:52 AM UTC
Resolved
Oct 05, 2022, 08:36 AM UTC
Duration
20h 44m
Detected by Pingoru
Oct 04, 2022, 11:52 AM UTC

Update timeline

  1. resolved Oct 05, 2022, 04:31 PM UTC

    A regression occurred between October 4, 13:52 CET and October 5, 10:36 CET, and caused custom machine learning model predictions to be ignored for some Implio teams, and under certain conditions. When those conditions were met, some moderation decisions during that timeframe were incorrect. - Affected *teams* were those with only one custom machine learning model (teams with multiple custom machine models for multiple languages were *not* affected). - Affected *items* were those with a language prediction (those without a language prediction were *not* affected). - Affected *rules* were those referring to the custom machine learning model prediction, exposed via the `$ml.status` variable (other rules were *not* affected). This regression was a side-effect of an improvement made to the platform. It escaped our tests and monitoring. After patching the regression, we've added the missing monitoring rules, so a similar incident would be picked up by our monitoring moving forward. We apologize for the impact this may have had on your moderation results. Please reach out to your Customer Success Manager for any question you may have.