Check incident

API 403 errors on May 26

Notice Resolved View vendor source →

Check experienced a notice incident on May 27, 2026, lasting —. The incident has been resolved; the full update timeline is below.

Started
May 27, 2026, 05:47 PM UTC
Resolved
May 27, 2026, 12:04 AM UTC
Duration
Detected by Pingoru
May 27, 2026, 05:47 PM UTC

Update timeline

  1. resolved May 27, 2026, 05:47 PM UTC

    On the evening of Tuesday, May 26, 2026, a code change caused a subset of API key requests to incorrectly return 403 Forbidden. We detected the issue after a partner report, reverted the change, and restored service within approximately 5 hours. We are deeply sorry for the disruption. This explains what happened and what we are doing to prevent recurrence. Impact The issue was live from Tuesday, May 26 at 8:04 PM ET to Wednesday, May 27 at 1:27 AM ET, a window of 5 hours and 23 minutes. During that time, API requests that create or update resources (POST and PATCH) returned 403 Forbidden for a subset of partners: partners who had their API key associated with a Console user of the roles standard or cost owner. Most partners were not affected. Read traffic (GET) and the underlying data were not affected; no records were created, modified, or lost as a result of this bug, and approved payrolls and scheduled payments continued to process normally. Root cause We deployed an authorization refactoring on our write endpoints. The change contained a bug that caused the new check to incorrectly reject legitimate write requests for some partners if their API key was tied to a Console user with a role of either standard or cost owner. These unintended errors were mixed with legitimately rejected traffic, which affected our detection time. The change was reverted. No further action is required from partners, and any request that failed with a 403 during the window can be viewed in API Logs and safely retried. What we are doing differently First, we will reintroduce the change behind a safer rollout, re-landing the additional write-path authorization check with additional automated tests for this API key provisioning setup and in more incremental stages. Second, we will tighten our deploy guardrails by restricting changes affecting authentication or authorization to adjusted hours, along with implementing more robust post-deploy monitoring for faster response. If you believe you saw a related issue outside the window above, or have questions about this incident, please reach out to us and reference this note.