Hosted Mender incident

Issues with Redis Cluster

Minor Resolved View vendor source →

Hosted Mender experienced a minor incident on October 6, 2025, lasting —. The incident has been resolved; the full update timeline is below.

Started
Oct 06, 2025, 08:26 AM UTC
Resolved
Sep 30, 2025, 01:30 PM UTC
Duration
Detected by Pingoru
Oct 06, 2025, 08:26 AM UTC

Update timeline

  1. resolved Oct 06, 2025, 08:26 AM UTC

    On September 25th, we updated the internal Redis Cluster to a new provisioning model, but it turned out that this new deployment was applied without a proper Kubernetes Priority Class being set. So we found out that during a new regular hosted Mender deployment, the pods with higher priority and "preempt = true" could preempt Redis pods to be scheduled faster. We applied a fix on the 30th of September, but during the previous 5 days we had at least four Redis Cluster issues, which lasted 1-2 minutes each. We apologize for the issues this incident might have caused to you.