Failed AWS deployments using OIDC authentication

Update timeline

1. investigating Nov 10, 2026, 08:58 AM UTC

   AWS deployments that authenticate with OIDC are failing for some organizations with the error "Not authorized to perform sts:AssumeRoleWithWebIdentity". We are investigating.

2. resolved Nov 10, 2026, 11:36 AM UTC

   Between 08:58 and 11:36 UTC, AWS deployments that authenticate with OIDC failed for a small number of organizations. A change in that morning's release caused this. No action was needed from affected customers. The issue is resolved, and failed deployments can be re-run. We apologize for the inconvenience.

3. investigating Nov 10, 2026, 01:47 PM UTC

   Postmortem: OIDC Tokens Switched to a Duplicate System User and AWS Rejected Them for 2.5 Hours Incident Description On 2026-11-10, AWS OIDC deployments for Allstate and Sophos failed with Not authorized to perform sts:AssumeRoleWithWebIdentity. env0/env0#23486 (APO-1319, released in v4.0.312) moved the OIDC system-user lookup from an Auth0 search to the credentials-service-prod-api-keys table. Some orgs have more than one Env0-OIDC-User in Auth0, created seconds apart by a create race in early 2025. The old code took the first Auth0 search result. The new code (findSystemUser, services/credentials-service/stores/api-key-store.ts:129) takes the oldest row. For 4 active orgs the oldest row is not the user their tokens used before, so the token sub changed. Allstate and Sophos pin sub in their AWS trust policies, so STS rejected every token. Timeline (UTC, Nov 10) 00:03: #23486 merges. 08:45: CD v4.0.312 starts (run 42785127357). credentials-service deploys 08:53:43 to 09:04:10. 08:57:10: Allstate’s original OIDC user mints its last token. 08:57:52: the duplicate mints its first. 08:57:23: same switch for Sophos. Its duplicate mints from 08:59:52. 08:59:55: first AssumeRoleWithWebIdentity failure in the SaaS agent logs (Sophos). 08:59 to 09:33: 2 more orgs switch users (1c7ec993, 13e6e9f5). Their deployments keep passing. 09:09: v4.0.312 run ends. 10:14: Allstate reports it in #envzero-allstate. No alert had fired. 10:43: thread in #engineering. 10:47: thread in #support-escalation-tier3. 10:51: credentials-service redeploy at v4.0.311 starts (run 42802640381). 10:52: root cause posted. 11:00: v4.0.311 is live. OIDC deployments still fail. 11:10: env0/env0#23563 (ATH-1781) opened with the permanent fix. 11:11: confirmed that the rollback did not help. About 11:11 to 11:21: app_metadata.apiKeyType set to oidc-duplicate on 10 duplicate users in 6 orgs. Each org keeps the user with the most Auth0 logins. 11:25: update posted. Tokens still use the duplicates, because the API Gateway Auth0 proxy caches the search for 1 hour. About 11:35: the 6 stale cache entries are refreshed one by one with Cache-Control: max-age=0. The whole stage cache is not flushed. 11:35:55: Allstate mints with its original user again. 11:36:36: Sophos. 11:36:57: 13e6e9f5. 11:37: resolved. No AssumeRoleWithWebIdentity failures for these orgs since. MTTD: 1 hour 15 minutes (08:59 → 10:14, from a customer report). Time to mitigate: 1 hour 22 minutes (10:14 → 11:36). Total impact: about 2 hours 37 minutes (08:59 → 11:36). What was the effect on production? 4 orgs had tokens minted by a different system user: Allstate (397 deployments), Sophos (126), 13e6e9f5 (14) and 1c7ec993 (3), from the token logs in auth0-actions-logs. 2 orgs failed: Allstate and Sophos. The other 2 do not pin sub, so their deployments passed. Sophos: at least 46 failed deployments in 18 environments, 08:59:55 to 11:30:00 (SaaS agent logs). Allstate: [TBC from Aurora deployment_logs: failed deployments and environments]. The SaaS agent logs show only 11, because most Allstate deployments do not run on the SaaS agent. Allstate also reported failing Azure deployments. [TBC: not confirmed from our data yet.] No backend errors. The token mint succeeded every time. What did the customer experience? AWS deployments failed at the credentials step with an STS error, with no change on their side. Allstate opened a bridge call with its leadership on it and asked for a vendor on the call and an RCA. What We Did “Good” Root cause found within 40 minutes of the report, from the Auth0 token logs: the minting user changed at 08:57, at the same time as the deploy. The mitigation changed only Auth0 metadata on 10 users and refreshed only 6 cache entries. A full cache flush would have dropped the cached token grants and risked Auth0 429s. A prod check across all 80 duplicate groups (30 OIDC, 50 Env0TerraformDeployer) showed that exactly one user per group has logins. That gives the permanent fix a safe rule. The customer got plain-language updates and a clear “you do not need to change anything”. What We Did “Bad” A customer found it, 75 minutes in. Alerts fire only on backend error logs. The mint succeeded and the failure happened in the customer’s STS call. The rollback did not roll back. The old code takes the first result of an unsorted Auth0 search, and Auth0 returns the most recently updated user first. For 2 hours the duplicate minted every token, so it became the most recently updated user and stayed first. v4.0.311 picked it too. The cache kept it broken after the data fix. The GET /auth0/{proxy+} proxy (shared/resources/api-auth0-proxy.yml) caches searches for 3600 seconds, so the corrected search was not used until the entries were refreshed. We could not see the minting user in our own logs. The OIDC token minted line in credentials-service logs apiKeyId as ****. Only the Auth0 action logs showed which user minted. The PR named the risk but guarded the wrong case. #23486 says a duplicate OIDC user changes sub, and guards a missing row. It does not handle two existing rows for one org. Root Cause The 2025 create race left duplicate system users in 80 groups. That was harmless while the Auth0 search returned the same user each time. #23486 added a new tie-break, “oldest row wins”, without checking it against the user each org actually used. For 21 OIDC orgs the oldest row is not that user, and 4 of them minted tokens in the window. In plain words: an org had two identical-looking robot users. We always used one of them. The new code started using the other one, and the customer’s AWS only trusts the first. Culprit PR Trigger: env0/env0#23486 (APO-1319). Latent: duplicate system users from the 2025 create race, and an unsorted users[0] pick in get-or-create-system-user.ts. Fix: env0/env0#23563 (ATH-1781), approved, not merged. Earliest step where this should have been caught #23486, at review. The PR text named the duplicate-user risk. A count of orgs with more than one system-user row, and a check that the new pick equals the user that minted the last token, would have found 21 orgs before release. Action Items Immediate Merge #23563 before the next credentials-service deploy. Without it, v4.0.313 picks the duplicates again. Owner: TBD. Clean up the duplicate system users: 30 orgs with duplicate Env0-OIDC-User and 50 with duplicate Env0TerraformDeployer. Keep the user with logins, mark the others in Auth0 and in the api-keys table. Owner: TBD.Short term Alert when deployments in several orgs fail with the same error in a short window (for example AssumeRoleWithWebIdentity, AADSTS). Owner: TBD. Post-deploy OIDC check: after each credentials-service deploy, compare the minting user per active OIDC org with the user before the deploy, and fail the deploy on a change. Owner: TBD. Log a hash of the minting user id in OIDC token minted instead of ****, so a change of user shows up in our logs and can be alerted on. Owner: TBD.Systemic When a PR names a risk, its guard and its tests cover that risk. APO-1319 named the duplicate-user risk but only guarded a missing row. Owner: TBD.