Astronomer experienced a major incident on May 20, 2025 affecting Deployment Management and Cloud Image Repository, lasting 1h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 20, 2025, 04:02 PM UTC
A small subset of customers have reported 403 errors when running the astro deploy command to deploy a new image. We are actively investigating this issue. If you are experiencing these errors, we encourage you to contact support and include the login command you used, astro cli and docker versions, and any log messages.
- monitoring May 20, 2025, 04:52 PM UTC
We've implemented a mitigation for this issue and the affected clusters should see successful image pushes. We will continue to monitor for additional errors.
- resolved May 20, 2025, 05:25 PM UTC
We have determined that this error is caused by cached credentials which are no longer valid after an internal change in Astro to the image registry. The fix must be performed client-side (i.e. on the machine running `astro deploy`). If you experience this error, run `docker logout` for each Astro registry that this machine has cached credentials for. By default, credentials are stored in ~/.docker/config.json, and if you are using this default setting, the following bash script will identify cached credentials and run docker logout for those that correspond to Astro registries. for domain in $(grep 'registry.astronomer.run' ~/.docker/config.json | awk '{print $1}' | tr -d '":' | sort | uniq); do docker logout "$domain" done