implio experienced a major incident on September 9, 2023 affecting implio core, lasting 1h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 09, 2023, 06:30 PM UTC
We are currently investigating this issue.
- identified Sep 09, 2023, 06:37 PM UTC
We are experiencing an issue moderating incoming items since 18:16 UTC: - Synchronous moderation endpoint: moderation requests denied with HTTP 500, - Asynchronous moderation endpoint: delayed processing. We are working to resolve this issue as soon as possible.
- monitoring Sep 09, 2023, 06:56 PM UTC
An internal service had issue starting while scaling up. The platform has been scaled manually.
- resolved Sep 09, 2023, 08:02 PM UTC
Implio has been running stable for over an hour. The incident is now considered as resolved. We will be investigating the root cause of the issue on Monday. Until then, the platform is running overprovisioned to prevent another potential incident.
- postmortem Sep 11, 2023, 01:58 PM UTC
The failure occurred in a service responsible for machine learning inference. It was attributed to memory starvation due to an imbalance between CPU and memory. To prevent future occurrences of this problem, we have: * Rebalanced the resources allocated to this service and increased memory significantly; * Removed resource limits, allowing pods to burst in CPU/memory usage while new pods are being started; * Added more monitoring metrics to better track memory usage. We sincerely apologize for any inconvenience this may have caused.