implio experienced a critical incident on June 9, 2023, lasting 12m. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jun 09, 2023, 04:14 AM UTC
The Implio platform has experienced issues in one of its internal services on 2023/06/09 between 02:50 and 03:02 UTC, with the following consequences: - Synchronous moderation endpoint: some of the requests were denied with HTTP 500 errors, - Asynchronous moderation endpoint: delayed processing. While the incident is closed, we are still investigating the root cause behind this issue and are actively monitoring the platform. We sincerely apologize for any inconvenience this may have caused.
- postmortem Jun 09, 2023, 05:29 AM UTC
Here’s an update on the incident and the follow-up actions we have taken: * We have pinpointed the issue to a memory exhaustion problem occurring on the core service responsible for evaluating rules. * We have doubled the amount of memory allocated to the service's pods, to avoid future occurrences of the problem. * There was a brief follow-up incident at around 05:02 UTC, before the fix was effective. We will continue to monitor the situation and further investigate durable ways to prevent similar problems in the future. Again, we sincerely apologize for any inconvenience this may have caused.