Slowness in processing of Tonkean items, actions and interfaces

Update timeline

1. investigating Apr 16, 2026, 08:33 AM UTC

   We're investigating a slowness in processing of Tonkean items, actions and interfaces

2. resolved Apr 16, 2026, 09:23 AM UTC

   This incident has been resolved.

3. postmortem Apr 16, 2026, 01:51 PM UTC

   On April 15, 2026 at 19:00 \(UTC\+3\), a configuration issue triggered a loop within an automation module, causing repeated item creation and excessive system activity. This led to a significant increase in load and degraded platform performance. Customers experienced elevated latency, including slower interface loading, delays when progressing through processes, and slower execution of automated actions. No data loss occurred. The issue was identified within approximately one hour. Mitigation actions included stopping the loop and clearing overloaded processing queues. Full service was restored for all customers by April 16 at 12:23 \(UTC\+3\). The root cause was a trigger configuration that continuously re-executed under certain conditions, resulting in unbounded item creation and system load. To prevent recurrence, we are implementing safeguards including rate limits on actions and item creation, defined execution quotas at the module and solution level, and automatic halting of processes when limits are exceeded with visibility in execution history.