Tonkean incident
Slowness in processing of Tonkean items, actions and interfaces
Tonkean experienced a minor incident on April 16, 2026 affecting Workflows Runtime and User Interfaces (Forms, Item Interfaces, Workspace Apps, Business Reports), lasting 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 16, 2026, 08:33 AM UTC
We're investigating a slowness in processing of Tonkean items, actions and interfaces
- resolved Apr 16, 2026, 09:23 AM UTC
This incident has been resolved.
- postmortem Apr 16, 2026, 01:51 PM UTC
On April 15, 2026 at 19:00 \(UTC\+3\), a configuration issue triggered a loop within an automation module, causing repeated item creation and excessive system activity. This led to a significant increase in load and degraded platform performance. Customers experienced elevated latency, including slower interface loading, delays when progressing through processes, and slower execution of automated actions. No data loss occurred. The issue was identified within approximately one hour. Mitigation actions included stopping the loop and clearing overloaded processing queues. Full service was restored for all customers by April 16 at 12:23 \(UTC\+3\). The root cause was a trigger configuration that continuously re-executed under certain conditions, resulting in unbounded item creation and system load. To prevent recurrence, we are implementing safeguards including rate limits on actions and item creation, defined execution quotas at the module and solution level, and automatic halting of processes when limits are exceeded with visibility in execution history.