Hosted Mender experienced a major incident on November 19, 2025 affecting Hosted Mender US, lasting 2h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Hosted Mender US
Update timeline
- investigating Nov 19, 2025, 05:08 AM UTC
We just upgraded the hosted Mender US cluster to Kubernetes v1.33 with a blue/green approach. The new cluster (blue) right after the switch is experiencing issues with the rate limits for some customer. We are investigating the issue.
- identified Nov 19, 2025, 06:12 AM UTC
We rolled back to the old green cluster, still having the same issue.
- monitoring Nov 19, 2025, 06:12 AM UTC
We found the root cause, a fix has been implemented, and we're monitoring the results.
- resolved Nov 19, 2025, 07:55 AM UTC
This incident has been resolved. However, a rate limit hot fix has been implemented, so we will schedule a new maintenance window soon, to apply the definitive fix.