Is Pulumi down?

We are investigating an issue affecting customers who sign in to Pulumi Cloud with a Bitbucket identity or use Bitbucket-connected accounts. Beginning at approximately 06:00 UTC on June 11, requests to refresh Bitbucket authentication tokens began failing. Affected users may experience failed sign-ins with Bitbucket, as well as intermittent errors in the Pulumi Console and API after their current Bitbucket session token expires (these tokens expire every two hours). Integrations that rely on Bitbucket credentials may also be affected. Our investigation indicates the failures originate from Bitbucket's OAuth token service rather than Pulumi infrastructure, and we have engaged Atlassian support. Customers using GitHub, GitLab, email, or SSO identities are unaffected. Workaround: Customers whose Pulumi account has an additional sign-in method (GitHub, GitLab, email/password, or SSO) can continue to sign in using that method. We will post updates as the investigation progresses.

Atlassian has acknowledged an incident affecting Bitbucket OAuth login, which is the cause of the Bitbucket sign-in and authentication failures affecting Pulumi Cloud. Their incident is being tracked at: https://bitbucket.status.atlassian.com/incidents/h51sr145gsmx The issue is external to Pulumi, and no action is required from Pulumi customers. Until Atlassian resolves their incident, customers signing in with a Bitbucket identity may continue to experience failed sign-ins and intermittent Console and API errors after their current session token expires. Customers using GitHub, GitLab, email, or SSO identities remain unaffected. Workaround: Customers whose Pulumi account has an additional sign-in method (GitHub, GitLab, email/password, or SSO) can continue to sign in using that method.

Atlassian has deployed a fix for their incident affecting Bitbucket OAuth login, and we have confirmed that Bitbucket authentication requests from Pulumi Cloud are succeeding again. Error rates have returned to normal levels and sign-in with Bitbucket is working.

The incident has now been resolved and the service is operating normally for all customers.

Investigating elevated API error rates and response times, particularly with ESC endpoints

A fix has been deployed and error rates and response times have recovered. We are continuing to monitor.

The incident has been resolved.

Some Pulumi Cloud organizations experienced elevated errors when decrypting ESC environments. A subset of environment revisions were affected, and those revisions failed to decrypt 100% of the time during the window. Overall, 3.32% of decrypt-attempt requests across Pulumi Cloud failed due to this issue. At the start of the window, we rolled out a change that prevented secret ciphertext copied from an ESC environment from being decrypted in other environments. Customers who had previously copied ciphertext between environments were affected. Users saw a message: “decryption failed. If this secret's ciphertext was copied from a different environment, copy and paste the plaintext instead.” This change was intended as a security improvement, but the customer impact was higher than expected, so the change was rolled back. The ESC team has conducted an internal postmortem to understand the misjudged customer impact, improve our responsiveness to scenarios like this, and created a plan to launch this security improvement without customer disruption.

Users authenticating with Bitbucket are currently unable to sign in to Pulumi Cloud or use the Pulumi Cloud API. This affects all operations that rely on Bitbucket authentication, including CLI usage and API calls. Other authentication methods are not affected. We have identified the cause of the issue and are actively working on a fix. We will provide an update once the fix has been deployed.

We have started deploying a fix for this issue. We will provide an update once the deployment is complete.

The fix has been deployed and we are seeing recovery. We are continuing to monitor to confirm the issue is fully resolved.

This incident has been resolved.

We are aware of reduced response times across Pulumi Cloud APIs. We have identified the cause and are working on a fix

This incident has been resolved.