Arcade.dev Outage History

This incident has been resolved.

The issue has been resolved. The issue stemmed from the addition of a new features allowing users to belong to multiple Arcade projects, and the chat application not using the newest APIs.

We have worked around the issue and have bug reports open with our identity provider to ensure this won't happen again.

We determined that because of recently adding thousands(!) of tools, one of our production components required a longer than expected startup process, meaning its health check endpoint was unavailable before the timeout expired. As a result, orchestrator was constantly restarting nodes, leading to poor availability of this critical component. Going forward, we plan to move these tasks out of component startup so this doesn't happen again and we can continue to support even larger numbers of tools.

From 7:30pm PST to 8:30pm PST users were unable to log into the dashboard. Non-dashboard related API calls were uninterrupted.

From 18:00 PST on July 7th, to 06:00 PST on July 8th, there was a partial outage for new end-users authenticating into Arcade tools for the first time. Existing OAuth connections were not effected. We have since corrected the issue.

Today between 11:48 Pacific and 13:46 Pacific we had a partial outage. New tool authorizations failed. Existing tokens were unaffected.

New users who attempted to create new accounts on Arcade.dev without providing company information were prevented from registering. This has been resolved.

A small number high-volume customers were impacted yesterday when we erroneously prevented them from authenticating tools for a percentage of their users. The issue was due to a bad cache configuration. We have reverted our changes and tools now works as expected.

X API was down, and therefore the X tools were unable to function