Slow processing of Tonkean items and access to the Tonkean app

Update timeline

1. investigating Apr 15, 2026, 05:32 PM UTC

   We've seen slowness in processing of Tonkean items. We're investigating the root cause and a possible fix.

2. investigating Apr 15, 2026, 07:33 PM UTC

   We are continuing to investigate this issue.

3. identified Apr 15, 2026, 08:32 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Apr 15, 2026, 09:11 PM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Apr 15, 2026, 09:16 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Apr 15, 2026, 09:22 PM UTC

   This incident has been resolved.

7. postmortem Apr 16, 2026, 01:52 PM UTC

   On April 15, 2026 at 19:00 \(UTC\+3\), a configuration issue triggered a loop within an automation module, causing repeated item creation and excessive system activity. This led to a significant increase in load and degraded platform performance. Customers experienced elevated latency, including slower interface loading, delays when progressing through processes, and slower execution of automated actions. No data loss occurred. The issue was identified within approximately one hour. Mitigation actions included stopping the loop and clearing overloaded processing queues. Full service was restored for all customers by April 16 at 12:23 \(UTC\+3\). The root cause was a trigger configuration that continuously re-executed under certain conditions, resulting in unbounded item creation and system load. To prevent recurrence, we are implementing safeguards including rate limits on actions and item creation, defined execution quotas at the module and solution level, and automatic halting of processes when limits are exceeded with visibility in execution history.