Astronomer incident

Permissions silently dropped from custom roles

Major Resolved

Astronomer experienced a major incident on September 17, 2026, lasting 3h 56m. The incident has been resolved; the full update timeline is below.

Started
Sep 17, 2026, 05:54 PM UTC
Resolved
Sep 17, 2026, 09:50 PM UTC
Duration
3h 56m
Detected by Pingoru
Sep 17, 2026, 05:54 PM UTC

Update timeline

  1. identified Sep 17, 2026, 05:54 PM UTC

    We have identified a regression affecting custom roles due to a change rolled out today. This resulted in permissions being dropped from custom roles silently. We have received reports of deploys linked to tokens using custom roles not working. Until the fix has been rolled out, deploy failures can be mitigated by re-adding the `deployment.deploys.create` permission to your custom role. You may also manually restore any other permissions that were silently dropped. We are working on fixing this regression for all affected custom roles.

  2. monitoring Sep 17, 2026, 07:29 PM UTC

    All custom roles have been restored to their previous functionality and full operation has been restored.

  3. resolved Sep 17, 2026, 09:50 PM UTC

    No further issues have been found, so this is being marked as resolved.