Tonkean Outage History

On April 15, 2026 at 19:00 \(UTC\+3\), a configuration issue triggered a loop within an automation module, causing repeated item creation and excessive system activity. This led to a significant increase in load and degraded platform performance. Customers experienced elevated latency, including slower interface loading, delays when progressing through processes, and slower execution of automated actions. No data loss occurred. The issue was identified within approximately one hour. Mitigation actions included stopping the loop and clearing overloaded processing queues. Full service was restored for all customers by April 16 at 12:23 \(UTC\+3\). The root cause was a trigger configuration that continuously re-executed under certain conditions, resulting in unbounded item creation and system load. To prevent recurrence, we are implementing safeguards including rate limits on actions and item creation, defined execution quotas at the module and solution level, and automatic halting of processes when limits are exceeded with visibility in execution history.

On April 15, 2026 at 19:00 \(UTC\+3\), a configuration issue triggered a loop within an automation module, causing repeated item creation and excessive system activity. This led to a significant increase in load and degraded platform performance. Customers experienced elevated latency, including slower interface loading, delays when progressing through processes, and slower execution of automated actions. No data loss occurred. The issue was identified within approximately one hour. Mitigation actions included stopping the loop and clearing overloaded processing queues. Full service was restored for all customers by April 16 at 12:23 \(UTC\+3\). The root cause was a trigger configuration that continuously re-executed under certain conditions, resulting in unbounded item creation and system load. To prevent recurrence, we are implementing safeguards including rate limits on actions and item creation, defined execution quotas at the module and solution level, and automatic halting of processes when limits are exceeded with visibility in execution history.