Tonkean incident

403 Forbidden Errors Detected on Specific Endpoints – Under Investigation

Major Resolved View vendor source →

Tonkean experienced a major incident on November 25, 2025 affecting User Interfaces (Forms, Item Interfaces, Workspace Apps, Business Reports) and Workflow Builder and 1 more component, lasting 18m. The incident has been resolved; the full update timeline is below.

Started
Nov 25, 2025, 08:51 AM UTC
Resolved
Nov 25, 2025, 09:09 AM UTC
Duration
18m
Detected by Pingoru
Nov 25, 2025, 08:51 AM UTC

Affected components

User Interfaces (Forms, Item Interfaces, Workspace Apps, Business Reports)Workflow BuilderUser Interface Builder (Forms, Item Interfaces, Workspace Apps, Business Reports)Workflow Runtime History

Update timeline

  1. investigating Nov 25, 2025, 08:51 AM UTC

    We are currently investigating this issue.

  2. resolved Nov 25, 2025, 09:09 AM UTC

    This incident has been resolved.

  3. postmortem Nov 27, 2025, 10:07 AM UTC

    **Root Cause Analysis: Partial API Service Disruption – November 25, 2025** On November 25, 2025, at 07:48 UTC, some users began experiencing issues in parts of the Tonkean platform. These included problems with the main menu, unexpected logouts with a “Board not found” error, and failures when connecting to integrations. 09:08 UTC, the incident was resolved, and all services were restored. The root cause was misconfigured API servers. Some of our backend systems had incorrect internal permissions. This caused specific requests routed to that server to be rejected, resulting in the observed errors. Restarting the service removed the faulty component and replaced it with a properly configured one. This issue only affected users whose requests were routed to the problematic server, resulting in inconsistent but frustrating impacts. No data was lost, and all services were fully restored within 80 minutes. To prevent similar problems in the future, we are adding automated checks that verify internal permissions are correctly set whenever services start. These checks will also include self-healing logic to fix misconfigurations when possible automatically. Our systems are currently stable and are being closely monitored.