Arcade.dev experienced a critical incident on November 6, 2025 affecting Engine API, lasting 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 06, 2025, 11:20 PM UTC
We are currently investigating an issue with our production deployments. We are attempting to rollback to a stable version. Investigation of the issue continues.
- investigating Nov 06, 2025, 11:22 PM UTC
We are continuing to investigate this issue.
- monitoring Nov 06, 2025, 11:30 PM UTC
Our prod services are now recovering. We are continuing to monitor and have identified the root cause of the issue.
- resolved Nov 06, 2025, 11:54 PM UTC
We determined that because of recently adding thousands(!) of tools, one of our production components required a longer than expected startup process, meaning its health check endpoint was unavailable before the timeout expired. As a result, orchestrator was constantly restarting nodes, leading to poor availability of this critical component. Going forward, we plan to move these tasks out of component startup so this doesn't happen again and we can continue to support even larger numbers of tools.