Tonkean incident
High errors rate in workflow execution + slowness in app loading
Resolved after 1 hour 17 minutes, affecting Workflows Runtime and User Interfaces (Forms, Item Interfaces, Workspace Apps, Business Reports) and 1 more, on June 25, 2026.
On June 24–25, 2026, between approximately 22:00 UTC on June 24 and 19:37 UTC on June 25, users experienced slowness and failures when opening certain views that rely on connected data sources. Fields linked to those data sources did not populate, and in some cases, views failed to load. We recognize the disruption this caused and appreciate your patience. The issue was first reported at 23:10 UTC on June 24. By 23:14 UTC, our engineering team had identified the cause: a separate project was generating an unusually large number of slow-running background queries that overwhelmed our shared search service, causing failures across connected data source fields. Service was confirmed restored by 19:37 UTC on June 25. No data was lost. The primary driver was a configuration error in a separate project. A connected data source had grown significantly, and due to the misconfiguration, every record in that source was treated as newly changed on every sync — causing the same data to be re-processed continuously and flooding the shared service. An existing safeguard that would have limited the impact had also been removed the prior day due to an unrelated bug. We corrected the misconfiguration to stop the excessive re-processing, and we are restoring the removed safeguard. Going forward, we are adding monitoring to detect and alert on this type of overload early, improving how we handle large data-source syncs so they do not flood shared services, and reviewing configuration defaults to prevent similar errors. We remain committed to the reliability of the Tonkean platform and to ensuring this type of incident does not recur.
Affected components
Update timeline
- investigating Jun 25, 2026, 06:57 PM UTC
We are seeing higher error rates and slow workflow execution times in production. We are investigating the root cause.
- resolved Jun 25, 2026, 08:15 PM UTC
All workflows and interfaces are back to normal.
- postmortem Jul 01, 2026, 09:24 AM UTC
On June 24–25, 2026, between approximately 22:00 UTC on June 24 and 19:37 UTC on June 25, users experienced slowness and failures when opening certain views that rely on connected data sources. Fields linked to those data sources did not populate, and in some cases, views failed to load. We recognize the disruption this caused and appreciate your patience. The issue was first reported at 23:10 UTC on June 24. By 23:14 UTC, our engineering team had identified the cause: a separate project was generating an unusually large number of slow-running background queries that overwhelmed our shared search service, causing failures across connected data source fields. Service was confirmed restored by 19:37 UTC on June 25. No data was lost. The primary driver was a configuration error in a separate project. A connected data source had grown significantly, and due to the misconfiguration, every record in that source was treated as newly changed on every sync — causing the same data to be re-processed continuously and flooding the shared service. An existing safeguard that would have limited the impact had also been removed the prior day due to an unrelated bug. We corrected the misconfiguration to stop the excessive re-processing, and we are restoring the removed safeguard. Going forward, we are adding monitoring to detect and alert on this type of overload early, improving how we handle large data-source syncs so they do not flood shared services, and reviewing configuration defaults to prevent similar errors. We remain committed to the reliability of the Tonkean platform and to ensuring this type of incident does not recur.
Your product runs on more than Tonkean
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services