Hosted Mender Outage History

Started Oct 08, 2026, 06:15 PM UTC· resolved Oct 08, 2026, 06:53 PM UTC

1. identified Oct 08, 2026, 06:15 PM UTC

   Some users get an "An error occured." message when they open Deployments > Finished. It happens on smaller screens and in browser windows that aren't maximized. We have found the cause, a display issue in the web interface, and are working on a fix. Devices, deployments and the API are not affected: deployments keep running and finishing as normal. Workaround: Make the browser window wider, or zoom out (Ctrl/Cmd and -), until the page loads.

2. monitoring Oct 08, 2026, 06:51 PM UTC

   We rolled back the release to quickly solve the issue. We're monitoring the results.

3. resolved Oct 08, 2026, 06:53 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 15, 2026, 07:30 PM UTC· resolved Sep 15, 2026, 08:18 PM UTC

1. investigating Sep 15, 2026, 07:30 PM UTC

   We are currently investigating this issue which has been reported by customers.

2. identified Sep 15, 2026, 07:44 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 15, 2026, 07:53 PM UTC

   A fix has been implemented and we're monitoring the results.

4. resolved Sep 15, 2026, 08:18 PM UTC

   This incident has been resolved.

5. postmortem Sep 16, 2026, 08:35 AM UTC

   ## Executive Summary We renew the signing key for our apt repositories every two years. The renewal on 2026-09-15 produced a key that Debian 13 \(trixie\) rejects. For about half a day, anyone installing Mender packages on Debian 13 got "`the repository is not signed`" and could not install anything. Existing devices were unaffected. Debian 12 and older were unaffected. No package was altered and nothing was exposed; this was a verification failure, not a security breach. ## What happened A signing key carries a signature saying "`this key is valid`". Renewing the key is supposed to extend that statement. The tool we use replaces it instead, so the renewed key effectively said "`valid as of today`" with no record of having been valid before. Debian 13 shipped a stricter signature checker. It asks whether the key was valid _at the moment the repository was signed_. Our repositories were last signed before the renewal, so the answer was no and it refused them. Debian 12 and older do not ask that question, which is why everything looked fine to us. ## Impact * Debian 13 users could not install or update Mender packages. ## Resolution We rebuilt the key so it carries both its old and its new validity statements. It is the same key with the same fingerprint, so nothing changed for existing devices, and it now satisfies both the old and the new checkers. Verified on Debian 12 and Debian 13, against every repository. **If you applied a workaround, re-fetch the key.** Some users unblocked themselves by fetching the key from `https://downloads.mender.io/repos/device-components/gpg`. During the incident that URL served the old key, which expires on **2026-10-01**, so those installs will start failing again in October. That URL now serves the corrected key, so fetch it once more: ``` curl -fsSL https://downloads.mender.io/repos/device-components/gpg \ | sudo gpg --dearmor -o /usr/share/keyrings/mender.gpg` ``` The expiry should now read 2028-09-14: ``` gpg --show-keys /usr/share/keyrings/mender.gpg ``` Adjust the output path if you installed the key somewhere else; the official docs place it at `/etc/apt/trusted.gpg.d/mender.asc`. No change to your `sources.list` is needed, and devices already running Mender need no action. ## Why we did not catch it 1. We have an internal renewal and verification runbook which doesn't include Debian 13, so we didn't catch it. 2. The repositories kept serving. Only clients on a newer distribution saw a problem, and we had no monitoring for that. ## Actions to avoid this issue the next time * We updated the internal runbook to also include Debian 13 and a note to check for future distributions. * We added a verification pipeline to the renewal process. ## Apology We are sorry. This was our mistake, and it blocked people from installing our software on a current, supported version of Debian. We have updated our internal renewal process to test against current Debian releases, and added a verification pipeline so this is checked automatically the next time. We did not find this ourselves. A customer did, and reported it. We are grateful for that report, which was precise enough that we could reproduce the problem within minutes, but catching this should not have been their job.

Read the full incident report →

Started Aug 30, 2026, 04:39 PM UTC· resolved Aug 30, 2026, 06:20 PM UTC

1. identified Aug 30, 2026, 04:39 PM UTC

   The latest release to hosted Mender on 2026-08-26T12:09Z introduced more optimized indexing of device inventory which had an unintended side effect of denying devices with array-type inventory data.

2. monitoring Aug 30, 2026, 04:44 PM UTC

   We have rolled back the affected service (inventory) to restore functionality for the affected devices. We are continuing to monitor the situation.

3. resolved Aug 30, 2026, 06:20 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 14, 2026, 03:42 PM UTC· resolved Jul 14, 2026, 06:38 PM UTC

1. identified Jul 14, 2026, 03:42 PM UTC

   An upstream provider notified that they are experiencing elevated error accessing API on EU region. It's possible that the Mender devices are unable to pull artifacts and deployments might fail. We're monitoring the upstream incident status.

2. resolved Jul 14, 2026, 06:38 PM UTC

   The upstream provider reported their incident is resolved.

Read the full incident report →

Started Jul 09, 2026, 06:08 PM UTC· resolved Jul 10, 2026, 11:21 AM UTC

1. investigating Jul 09, 2026, 06:08 PM UTC

   We are currently investigating the root cause of the issue.

2. monitoring Jul 10, 2026, 06:33 AM UTC

   We found a possible root cause with the new intermediate certificate of https://eu.hosted.mender.io with a different size. A tested workaround is to increase the mbedTLS heap size in your application's prj.conf: CONFIG_MBEDTLS_HEAP_SIZE=98304.

3. resolved Jul 10, 2026, 11:21 AM UTC

   This incident has been resolved by tuning the mbedTLS heap size.

4. postmortem Aug 28, 2026, 08:24 AM UTC

   ## **Executive Summary** An issue was identified where microcontroller \(MCU\) devices, specifically those running Zephyr OS and `mender-mcu` on constrained hardware like the ESP32-S3, lost connection to the hosted Mender EU server \(`eu.hosted.mender.io`\). Specifically, devices failed during the initial TLS handshake with error `-0x2700`, preventing Over-The-Air \(OTA\) updates and inventory reporting. The investigation confirmed that the root cause was **insufficient client-side memory \(mbedTLS heap\)** when processing the server's RSA-4096 certificate chain, rather than a certificate validity or time-synchronization issue. To recover devices already deployed in the field without requiring a physical intervention, the engineering team deployed a **server-side mitigation by switching the production TLS certificate authority to Google Trust Services \(GTS Root R4\)**. This restored connectivity for affected field devices. ## **Impact** Customers using MCU devices targeting the EU instance were not able to perform updates from the 9th to the 15th of July. ## **Technical Root Cause** The hosted Mender EU endpoint utilized a cross-signed Let’s Encrypt certificate hierarchy ending in an RSA-4096 leaf and root certificate. From the 4th of June, letsencrypt \(EU\) started using a different chain from the same root CA, with one more intermediate. _Jan – early Jun 2026 │ leaf → R12/R13 → ISRG Root X1 --> 1 intermediate_ _Jun 4 2026 → now │ leaf → YR1 → ISRG Root YR → ISRG Root X1 --> 2 intermediates_ This change landed at the beginning of July to hosted Mender EU, after automated cert-bot renewal, and then the mender-mcu clients lost connection. During the TLS handshake, mbedTLS requires continuous heap memory to simultaneously hold: * The TLS input and output content buffers \(`SSL_MAX_CONTENT_LEN`\). * The parsed peer certificate chain \(RSA-4096\). * The parsed trust anchors. * The temporary mathematical buffers required to execute an RSA-4096 signature verification. On constrained MCU devices, the default mbedTLS heap configuration \(`CONFIG_MBEDTLS_HEAP_SIZE=40960`, or ~40 KB\) was exceeded. Our tTesting revealed that processing this specific RSA chain required a minimum heap threshold of ~41.5 KB \(42496 bytes\). When the memory allocation failed internally during signature verification, mbedTLS surfaced the out-of-memory exception as a general certificate trust failure \(flag 0x08: `BADCERT_NOT_TRUSTED`\). A mitigation was published, to increase the `CONFIG_MBEDTLS_HEAP_SIZE`, but of course this can’t be applied to devices which lost the connection to the server. ## **Resolution & Recovery** ### **Immediate Field Recovery \(Server-Side\)** Because deployed devices with broken OTA connectivity cannot receive a firmware update to increase their local heap size, engineering deployed a server-side resolution: * Switched the production certificate chain for [eu.hosted.mender.io](http://eu.hosted.mender.io) from Let's Encrypt to **GTS Root R4 \(Google Trust Services\)**. * The GTS Root R4 chain is already bundled within the existing MCU client trust stores and requires significantly lower memory overhead during verification, allowing already-deployed devices to successfully complete the handshake and re-establish OTA connectivity without a client-side patch. * A previous attempt to force a lighter CA Chain with ECDSA root CA failed; although the required heap was less than the new RSA root, it was still heavier than the old RSA root, so our internal validation failed. ### **Long term solutions** * We are implementing an end-to-end failover strategy covering both TLS infrastructure and domain name \(DNS\) routing. * We implemented periodic tests to emulate a MCU device connecting to both US and EU instances to early catch this kind of issues. ‌ We sincerely apologize for the disruption this caused to your fleets, this was a serious failure, and we understand the operational burden it placed on you. We are committed to the improvements described above and to ensuring that changes in upstream certificate infrastructure never again silently break connectivity for constrained devices.

Read the full incident report →

Started Jun 24, 2026, 12:10 PM UTC· resolved Jun 24, 2026, 12:32 PM UTC

1. investigating Jun 24, 2026, 12:10 PM UTC

   We are currently investigating this issue.

2. identified Jun 24, 2026, 12:18 PM UTC

   The issue has been identified: there was an issue with the Redis and NATS deployments. A fix has been implemented and we're restarting the impaired services.

3. monitoring Jun 24, 2026, 12:23 PM UTC

   The affected services have been restarted and we're monitoring the results.

4. resolved Jun 24, 2026, 12:32 PM UTC

   This incident has been resolved.

5. postmortem Jun 24, 2026, 01:34 PM UTC

   # Brief service disruption on Hosted Mender \(EU\) On June 24, between 12:04 and 12:09 UTC, some customers on our EU region experienced a short interruption. During this five minute window, devices may have had trouble connecting, and parts of the web interface and API may have returned errors. ## What happened While cleaning up a configuration issue, we removed an internal duplicated definition \(the same set of kubernetes manifests rendered twice from Kustomize\). Our automation interpreted this as a request to remove and reinitialize two background components that handle messaging and caching \(NATS and Redis\). They were rebuilt automatically, but for about five minutes they were unavailable, which caused the disruption. ## What we are doing about it The automation completed the reconciliation loop and redeployed the background components automatically. We verified that normal operation was restored and confirmed all services are healthy. We are changing how we make this type of change so that components are never removed before their replacement is ready. We apologize for the disruption.

Read the full incident report →

Started Jun 18, 2026, 04:30 PM UTC· resolved Jun 18, 2026, 07:00 PM UTC

1. resolved Jun 19, 2026, 11:56 AM UTC

   Incident Summary - 2026-06-18 What happened On June 18, between 18:46 UTC and approximately 19:00 UTC, some users may have experienced intermittent connectivity issues or elevated error rates when accessing Mender services from the EU cluster. The incident was triggered by a scheduled maintenance job running on the infrastructure hosting our services. This job - responsible for cleaning up unused container images across cluster nodes - ran simultaneously on all nodes and consumed a significant amount of CPU resources. This created resource contention with the ingress controller pods responsible for routing incoming traffic, causing them to become temporarily unresponsive to health checks and restart repeatedly. The repeated restarts reduced the number of healthy ingress endpoints below the threshold required to serve traffic across all availability zones, leading to degraded routing and elevated error rates for most of the requests. Resolution The ingress controller stabilized once the image cleanup job completed. This job is a new managed feature coming from the recent Kubernetes upgrades, and since it’s not a critical feature, we suspended it. Additional safeguards have been put in place to prevent ingress controller pods from being affected by resource pressure from unrelated workloads. What we are doing The image cleanup job has been permanently suspended. CPU resource limits have been tightened on the ingress layer to isolate it from competing workloads. Node pool capacity is being expanded with larger instances to provide additional headroom. We apologize for the disruption.

Read the full incident report →

Started Jun 17, 2026, 06:12 PM UTC· resolved Jun 17, 2026, 06:36 PM UTC

1. identified Jun 17, 2026, 06:12 PM UTC

   We've identified a regression affecting users who access Mender with a custom role (rather than the built-in administrator role). As a result, we have rolled back to the previous version. When a user with a custom role attempts to list software when creating a deployment, the request is rejected with There was an error while listing software forbidden by role-based access control Only users assigned a custom role were affected. Users with the built-in admin role were not affected. We apologize sincerely for this issue.

2. monitoring Jun 17, 2026, 06:17 PM UTC

   We rolled back the affected version and we're monitoring the results

3. resolved Jun 17, 2026, 06:36 PM UTC

   This incident has been resolved, the rollback worked and we'll apply the fix in the next release.

Read the full incident report →

Started Apr 08, 2026, 01:16 PM UTC· resolved Apr 08, 2026, 02:13 PM UTC

1. investigating Apr 08, 2026, 01:40 PM UTC

   We are currently investigating the issue

2. identified Apr 08, 2026, 01:44 PM UTC

   The issue has been identified and a fix is being implemented

3. monitoring Apr 08, 2026, 01:58 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 08, 2026, 02:13 PM UTC

   Mender UI was unavailable between 2026-04-08T13:16:15Z and 2026-04-08T13:42:09Z (26m) due to a breaking change to Google Analytics dependency leading to misconfiguration. The issue has been mitigated by temporarily disabling Google Analytics until a fix is deployed.

Read the full incident report →

Started Feb 20, 2026, 09:28 AM UTC· resolved Feb 20, 2026, 12:50 PM UTC

1. investigating Feb 20, 2026, 09:28 AM UTC

   We are currently investigating a performance issue reported by our monitoring system

2. investigating Feb 20, 2026, 09:51 AM UTC

   It's not currently possible to accept new devices. We're continuing to investigate the issue.

3. monitoring Feb 20, 2026, 11:33 AM UTC

   We identified an issue and applied a fix. We're monitoring the results. Now it's possible to accept new devices and the performance issue should be gone.

4. resolved Feb 20, 2026, 12:50 PM UTC

   This incident has been resolved.

5. postmortem Feb 26, 2026, 04:10 PM UTC

   # **Database Overload from Device Limit Migration Bug** **Date:** 2026-02-20 **Duration:** ~3 hours 25 minutes \(08:15 - 11:40 UTC\) **Severity:** High ## **Executive Summary** On February 20, 2026, the Hosted Mender platform experienced a critical service outage affecting device authentication and inventory operations. A change deployed as part of Mender v4.2.0-saas.2 failed to uniformly handle data inconsistencies in older tenant configurations. A specific call order of two independent backend endpoints in combination with scheduled cache invalidation uncovered a bug which caused a heavy increase in load on the database. The unexpected increase in load was beyond what the system is designed to handle which resulted in cascading errors and platform-wide degradation. ## **Impact** * **Duration:** Approximately 3 hours 25 minutes * **Scope:** Multi-tenant platform-wide degradation * **Affected Services:** Device authentication, device inventory * **User Experience:** Unable to accept new devices * **Business Impact:** Complete halt to device provisioning across the platform \(hosted Mender US only\) during incident window ## **Root Cause** In Mender v4.2.0-saas.2 we changed the definition of an “unlimited” device limit from 0 to -1 so the system would be able to represent limits that allow zero devices. This was done by introducing a database migration that migrated existing limits with the value 0 to have the value -1 instead and cleared the limits cache to ensure data would be collected fresh from the database post migration. Lastly, we were aware of a known edge case where certain tenants would not have a limit defined in the database and took steps to ensure consistent handling of this scenario post migration. This new version of Mender also included an internal endpoint that incorrectly set the cached device limit of a tenant to 0 in the case where a\) there was no limit in the cache from before and b\) there also was no limit in the database. This endpoint was overlooked in the steps mentioned above. When the internal endpoint was called after the cache was invalidated, but before any external endpoints that used device limits, the limit 0 was incorrectly cached for some tenants with a large number of devices and no device limit in the database. When the device authorization reprocessing logic was executed for these devices, the incorrectly cached limit caused a large amount of database queries to be executed in order to check if the limit had been exceeded \(something which is not necessary to check if the device limit is “unlimited”\). No matter the result of the check, a limit of 0 will always result in the device not being allowed to authorize with the system and devices will continuously retry in such a case, amplifying the issue manyfold until eventual and complete MongoDB resources exhaustion. ## **Timeline \(All times UTC\)** **2026-02-19** * **13:16** - Deployed Mender v4.2.0-saas.2 - _Root cause introduced_ **2026-02-20** * **~08:00** - Devices of the affected tenants started the authorization reprocessing process * **08:15** - A synthetic test failure alerted the On-call team * **08:20** - On-call investigated tenant configuration; Admin Panel queries failing with 499/504 due to DB exhaustion * **08:20** - Identified ongoing device authorization reprocessing consuming all database resources * **09:25** - Attempted to stop problematic queries * **10:30** - Discovered blocked queries still holding locks; initiated emergency database scaling * **10:40** - Database scaled; locks cleared; device acceptance partially restored * **11:55** - Cache for device-auth disabled * **11:00** - Added missing limits with value -1 \(unlimited\) in the database affected tenants * **11:40** - Service fully restored ## **What went wrong** 1. **Inadequate test coverage**The test coverage of the internal endpoint was inadequate as it didn’t verify that the correct value was used and cached in this scenario. 2. **Inadequate manual testing**Manual testing was performed, but not with a cache that was explicitly invalidated for this purpose. 3. **Uncontrolled Cascade**The device authorization reprocessing logic had a snowball effect on the platform. ## **Action Items** * Resolve the issue where limits who are intended to be “unlimited” can be incorrectly cached as 0 by this internal endpoint. * Update the device authorization reprocessing logic to not execute unnecessary database queries if the limit is 0. * Review and improve test coverage of the affected endpoints. ## **Conclusions** We want to sincerely apologize for the service disruption you experienced on February 20, 2026. For over three hours, our platform was unable to process device authentication and inventory operations, preventing you from onboarding new devices and managing your fleet. We are committed to prevent this kind of disruption in the future.

Read the full incident report →

Started Nov 19, 2025, 05:08 AM UTC· resolved Nov 19, 2025, 07:55 AM UTC

1. investigating Nov 19, 2025, 05:08 AM UTC

   We just upgraded the hosted Mender US cluster to Kubernetes v1.33 with a blue/green approach. The new cluster (blue) right after the switch is experiencing issues with the rate limits for some customer. We are investigating the issue.

2. identified Nov 19, 2025, 06:12 AM UTC

   We rolled back to the old green cluster, still having the same issue.

3. monitoring Nov 19, 2025, 06:12 AM UTC

   We found the root cause, a fix has been implemented, and we're monitoring the results.

4. resolved Nov 19, 2025, 07:55 AM UTC

   This incident has been resolved. However, a rate limit hot fix has been implemented, so we will schedule a new maintenance window soon, to apply the definitive fix.

Read the full incident report →

Started Oct 31, 2025, 08:58 PM UTC· resolved Nov 03, 2025, 01:19 PM UTC

1. investigating Oct 31, 2025, 08:58 PM UTC

   We are receiving some reports of problems with webhooks not triggering for device provisioning. We are also receiving report of problems with AWS IoT integrations.

2. investigating Oct 31, 2025, 08:58 PM UTC

   Regarding the Webhook issue, a possible workaround is to re-create the failing webhook. We're still investigating the root cause.

3. identified Nov 03, 2025, 11:31 AM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Nov 03, 2025, 01:19 PM UTC

   We reverted a recent change that caused an unexpected behavior with the encryption algorithm, and we re-encrypted the secrets for the IoT Manager integration. Now the normal operation is restored.

5. postmortem Nov 06, 2025, 08:42 PM UTC

   **Abstract** On Monday, 27th of October, we released the Mender Server v4.1.0-saas.16 to hosted Mender US and EU. Among the many changes, there was also a change indirectly changing the cipher method for client side encryption of secrets in the IoT Manager database. The change replaced the deprecated Cipher Feedback \(CFB\) cipher mode with Counter \(CTR\) mode as suggested by the [Golang documentation](https://pkg.go.dev/crypto/[email protected]#NewCFBDecrypter). On October 31th, at about 8PM, we were alerted by multiple tickets opened by customers regarding webhooks not working for AWS IoT integration. The on-call team then opened an [incident](https://mender.statuspage.io/incidents/g019zy922897). The engineering team then on Monday, 3rd of November, soon acknowledged the issue and found out the root cause. We briefly discussed how to solve the issue and we decided to rollback the IoT Manager service and re-encrypt the secret with the old algorithm for the affected customers. Two customers, however, already updated their config, because they recreated the webhook configuration, after being suggested by the Northern Tech team as a valid workaround, so the rollback affected their operation for a second time. We are really sorry for the inconvenience, and we are working to fix this process around the IoT Manager integration. **Incident Timeline \(UTC\)** * 2025-10-27 12AM - Mender Server v4.1.0-saas.16 released on hosted Mender EU and US * 2025-10-31 8PM - The Customer Engineer team alerted because of multiple ticket regarding failing IoT Integration * 2025-10-31 8:58PM - This incident has been opened * 2025-11-03 12AM - We reverted the IoT Manager version, decrypted the secret with the new cipher and re-encrypted it again with the old cipher, restoring the operation * 2025-11-03 11AM - Mender server v4.1.0-saas.17 was released to hosted Mender US and EU, including the revert commit for the new cipher, restoring the old one. **What went wrong** Multiple failure at multiple level: * we lack of IoT Manager upgrade tests; for this specific issue, unit and integration tests didn’t catch the issue because they are performing tests on new fresh data, encrypted with the new cipher; * we lack of Synthetic Tests on IoT Manager; * we suggested a workaround for restoring the situation as a first step, but then the rollback to the previous version caused another disruption to some customers. **Actions we decided to take to prevent this issue in the future** * Improve the logging and monitoring around the IoT Manager service * introduce new error log when webhooks are misbehaving and build metrics and alert based on the new log to catch issues faster * Introduce Synthetics tests to periodically assess the IoT Manager webhook functionality * Improve error handling by registering unsuccessful attempts to send webhooks * Register timestamp on secret update and creation, to easily understand the history of a secret * We still need to replace the outdated cipher, we will plan a non disruptive update * Introduce upgrade tests, to check that the IoT Manager service could work with both the old and the new version.

Read the full incident report →

Started Oct 22, 2025, 10:13 AM UTC· resolved Oct 22, 2025, 11:31 AM UTC

1. monitoring Oct 22, 2025, 10:13 AM UTC

   This is the same incident as https://mender.statuspage.io/incidents/v85h4c0whdtc, which is keeping track of it for the hosted Mender EU The issue has been identified. A migration triggered by an upgrade caused an index to be removed prematurely. This in turn caused data corruption. We have initiated a database restore and rolled back the upgrade. We apologize for the inconvenience.

2. monitoring Oct 22, 2025, 10:14 AM UTC

   The restore to 08:10:00 UTC has completed and the server is scaled back up. We will continue to monitor the situation.

3. resolved Oct 22, 2025, 11:31 AM UTC

   This incident has been resolved

Read the full incident report →

Started Oct 22, 2025, 08:56 AM UTC· resolved Oct 22, 2025, 11:32 AM UTC

1. investigating Oct 22, 2025, 08:56 AM UTC

   We noted a spike in API error metrics; we are investigating the issue.

2. investigating Oct 22, 2025, 09:07 AM UTC

   We are continuing to investigate this issue.

3. identified Oct 22, 2025, 09:20 AM UTC

   The issue has been identified. A migration triggered by an upgrade caused an index to be removed prematurely. This in turn caused data corruption. We have initiated a database restore and rolled back the upgrade. We apologize for the inconvenience.

4. monitoring Oct 22, 2025, 09:44 AM UTC

   The restore to 08:10:00 UTC has completed at 09:28:49 UTC and the server is scaled back up. We will continue to monitor the situation.

5. resolved Oct 22, 2025, 11:32 AM UTC

   This incident has been resolved

6. postmortem Dec 11, 2025, 08:26 PM UTC

   **Date:** October 22, 2025 **Duration:** 78 minutes \(08:10 - 09:28 UTC\) **Severity:** Major service disruption ‌ **Executive Summary** A database migration in release v4.1.0-saas.16 caused a complete failure of the Device Authentication service across US and EU hosted Mender clusters. The migration incorrectly deleted a critical uniqueness constraint during online operations, leading to database corruption that prevented service recovery. We restored service by performing a point-in-time database rollback, resulting in 78 minutes of data loss. **Customer Impact**: Device authentication was unavailable for 78 minutes. New device enrollments were blocked, and existing device operations may have been disrupted during this period. **Root cause** The new version contained a database migration to 2.0.1 for the Device Auth database, which was designed to replace a uniqueness constraint on device authentication records but executed the deletion and recreation as separate operations. During online migration, the window between index deletion and recreation allowed duplicate device entries to be created, corrupting the database state and preventing both forward migration completion and rollback. For this reason, the only viable solution was to rollback both the Mender Server version and the Database. ‌ **Resolution and recovery** With duplicate records preventing normal rollback procedures, we performed a point-in-time database restore to 08:10 UTC, with a safe timestamp before migration execution. This restored database integrity but resulted in permanent loss of all data created between 08:10 and 09:28. ‌ **Incident timeline \(UTC\)** * 08:35 AM - the new v4.1.0-saas.16 version was published and both hosted Mender US and EU started the automated upgrade * 08:40 AM - the upgrade failed and rolled back automatically to v4.1.0-saas.15, because the deviceauth service wasn’t able to complete the migration job. * 08:42 AM - the On-call team acknowledged a possible issue with the upgrade, in the meantime the deviceauth service and MongoDB were at 100% load, because of the missing index. * 09:16 AM - we decided to restore the MongoDB database to the Point-in-time with timestamp 08:10:00 AM and the restoration process started. * 09:28 AM - the MongoDB restoration process finished. ‌ **What went wrong** * **Migration Strategy**: The migration required an offline window or an atomic operation strategy, but this requirement was not identified during development or code review. * **Testing Gaps**: Pre-release testing did not simulate high-concurrency writing during the migration, failing to trigger the race condition found in production. * **Data loss**: We failed to export a snapshot of the corrupted state before the point-in-time retention window expired. ‌ **Action Items** * **Enhance Load Testing**: pre-release tests are not sufficient to really simulate the production environment, so to catch this issue in an early stage. We are planning to run load testing and chaos testing more often and extensibly to mitigate this risk. * **Update the rollback playbook**: mandate that a snapshot of the "corrupted" database state be taken immediately following a destructive Point-in-Time recovery to preserve data and to allow recovery of data if necessary. ‌ We sincerely apologize for the disruption to your operations and, specifically, for the data loss that occurred during the recovery window.

Read the full incident report →

Started Oct 20, 2025, 09:55 AM UTC· resolved Oct 20, 2025, 12:37 PM UTC

1. identified Oct 20, 2025, 09:55 AM UTC

   We identified an issue with an upstream cloud provider. Unfortunately, even our statuspage was affected and we could not open this status page before now. Among the issues, we identified: * issues reaching the hosted Mender API server * emails not being sent * UI login not possible. The incident started at about 07:50 UTC

2. identified Oct 20, 2025, 09:58 AM UTC

   The cloud provider is fixing their issues, we can see that most of the hosted Mender services are back to full operation. We're continuing to monitor the issue. The email service is still not working.

3. resolved Oct 20, 2025, 12:37 PM UTC

   This incident has been solved.

Read the full incident report →

Started Oct 08, 2025, 03:53 AM UTC· resolved Oct 08, 2025, 09:09 AM UTC

1. identified Oct 08, 2025, 03:53 AM UTC

   We are aware of an open incident with a Cloud Provider for the Artifacts Storage for both hosted Mender US and EU. You may experience slow artifact download or throttling. We're monitoring the upstream provider's status.

2. resolved Oct 08, 2025, 09:09 AM UTC

   The upstream provider solved their issue.

Read the full incident report →

Started Sep 30, 2025, 01:30 PM UTC· resolved Sep 30, 2025, 01:30 PM UTC

1. resolved Oct 06, 2025, 08:26 AM UTC

   On September 25th, we updated the internal Redis Cluster to a new provisioning model, but it turned out that this new deployment was applied without a proper Kubernetes Priority Class being set. So we found out that during a new regular hosted Mender deployment, the pods with higher priority and "preempt = true" could preempt Redis pods to be scheduled faster. We applied a fix on the 30th of September, but during the previous 5 days we had at least four Redis Cluster issues, which lasted 1-2 minutes each. We apologize for the issues this incident might have caused to you.

Read the full incident report →

Started Sep 26, 2025, 06:41 AM UTC· resolved Sep 26, 2025, 12:51 PM UTC

1. investigating Sep 26, 2025, 06:41 AM UTC

   Customers are reporting issues with the remote terminal (Deviceconnect Service) which is frequently dropping the remote connection. We are investigating this issue.

2. identified Sep 26, 2025, 06:43 AM UTC

   The issue has been identified and we're working on a solution.

3. monitoring Sep 26, 2025, 10:39 AM UTC

   We rolled back the Deviceconnect service to the previous known working version. We're monitoring the metrics and the results.

4. resolved Sep 26, 2025, 12:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 16, 2025, 01:58 PM UTC· resolved Sep 16, 2025, 04:07 PM UTC

1. investigating Sep 16, 2025, 01:58 PM UTC

   We are aware of an issue regarding the deployment service and an unusual amount of server errors. We are investigating the issue.

2. identified Sep 16, 2025, 03:18 PM UTC

   The issue has been identified. We scaled up both the MongoDB cluster and the backend services to cope with an unexpected load.

3. monitoring Sep 16, 2025, 03:30 PM UTC

   After scaling the cluster up, the APIs are back to the expected operations. We're continuing monitoring the cluster health.

4. resolved Sep 16, 2025, 04:07 PM UTC

   This incident has been resolved.

5. postmortem Sep 26, 2025, 04:21 PM UTC

   **Abstract** On Tuesday, September 16th, at around 01:50 UTC, a Mender deployment was created from a single tenant targeting about 16,000 devices with a very short polling interval. This caused a sudden load and memory spike in the Deployments Service, the Device Auth Service, and MongoDB. Unfortunately, the MongoDB cluster was unable to scale automatically at that time due to the sudden spike. As a result, we had to first deactivate the Deployments Service, abort the massive deployment, and then allow MongoDB to recover properly. Only then were we able to scale it up and restore full operations. We apologize for the disruption this incident caused. We know that hosted Mender can scale well under certain conditions, but in this case the scaling was too slow and we have to address this issue. **Incident Timeline** * 2025-09-16 01:52PM - The alert NodeHighNumberConntrackEntriesUsed was sent to the on-call operator. The immediate issue was MongoDB rolling back and deployments service and device-auth services restarting due to Out of Memory Kills * 2025-09-16 02:01PM - The root cause was identified: a massive amount of v2/deployments/next were being served \(around 2000 per minute, compared to an average of 20 per minute\) * 2025-09-16 02:10PM - we scaled the Deployments Service and the Device Auth Service, increasing memory by 200% * 2025-09-16 02:15PM - we attempted to apply per-API limits on the tenant causing the massive deployment, but this had no effect. * 2025-09-16 02:20PM - the MongoDB cluster remained unavailable due to repeated crashes across all three nodes. We decided to scale Deployments Service and the Device Auth Service down to 0 to allow MongoDB to recover. * 2025-09-16 02:40PM - MongoDB was fully restored, and we scaled the db up from M40 to M60. * 2025-09-16 02:52PM - MongoDB was successfully running at M60. * 2025-09-16 02:56PM - We marked the massive deployment operation as “finished” to prevent further disruption * 2025-09-16 03:00PM - Since the massive deployment had been aborted, we scaled the Deployments Service and the Device Auth Service back to full operation. * 2025-09-16 03:02PM - hosted Mender US service was fully operational again. * 2025-09-16 03:10PM - we notified the customer that we had aborted the massive deployment. We also prevented the MongoDB cluster from scaling back to M40; the new minimum size is now M50. **What went wrong** The Mender Server is currently managing more than 500.000 active devices, which usually poll the API server at long intervals \(hours or once per day\). Under these conditions, the Deployments Service typically serves about 30 deployments per minute on average, with peaks of around 1,000 per minute. In this case, however, a single deployment targeted about 16,000 devices with a polling interval of 5 minutes or less. As a result, all deployments started nearly simultaneously, leaving neither MongoDB nor the Kubernetes cluster enough time to scale up properly. Both MongoDB and the Deployments Service went out of memory. In short, hosted Mender cannot currently handle this type of sudden load, and we must address this issue. **What we did in the short term** * Disabled the massive deployment and coordinated with the customer to perform deployments in a way that is more optimized for the current cluster capacity. * Scaled up MongoDB, the Deployments Service, and the Device Auth Service to help prevent a recurrence. **What should we do in the long term** We already have a very basic rate limiting in the API server, but it must be improved to protect hosted Mender from usage patterns that could negatively impact other customers, and to allow the Kubernetes and the MongoDB cluster to scale up properly. We will commit to implementing a new set of rate limits.

Read the full incident report →

Started Aug 26, 2025, 05:37 PM UTC· resolved Aug 27, 2025, 09:46 AM UTC

1. investigating Aug 26, 2025, 05:37 PM UTC

   We are investigating the root cause of the device authorization backend issue.

2. identified Aug 26, 2025, 05:59 PM UTC

   The issue has been identified, and we have implemented a temporary fix. We are monitoring the temporary fix.

3. resolved Aug 27, 2025, 09:46 AM UTC

   This incident has been resolved.

4. postmortem Sep 01, 2025, 12:02 PM UTC

   **Abstract** On the 26th of August, at 05:01 PM UTC, we received multiple alerts regarding the hosted Mender EU cluster; in particular many 5xx errors and the pods for the device-auth service which was restarting. Because of this reason the hosted Mender cluster was not fully operational since the devices could not authenticate. The reason for the crash was multiple “out of memory” kills. The on-call team member increased the memory available for the pod, in order to avoid the restarts and restore the hosted Mender functionality. During the following day the backend team analyzed the issue and found that the cluster was receiving an unusual amount of requests to the endpoint `GET /api/management/v2/devauth/devices`: it appeared that a Python agent requested the same resource approximately 200 times per minute \(up from 6 requests per minute before\). We alerted the customer from which the requests were originating, which confirmed an unintended issue on their end and fixed the issue. **Incident timeline**: 26th of August * 5:01 PM - start seeing 5xx errors * 5:02 PM - mender-device-auth restarting * 5:49 PM - memory increased - incident finished 27th of August * about 9 AM - the customer was alerted * about 1 PM - the customer fixed the issue

Read the full incident report →

Started Aug 16, 2025, 05:29 AM UTC· resolved Aug 16, 2025, 08:16 AM UTC

1. investigating Aug 16, 2025, 05:29 AM UTC

   When you try to upgrade your subscription in the Billing page, you can't see the Checkout button and so you cannot upgrade your subscription. We're investigating the issue.

2. resolved Aug 16, 2025, 08:16 AM UTC

   We reverted a change for the billing page. Now the behavior should be the one before the version v4.1.0-saas.12 of hosted Mender.

Read the full incident report →

Started Aug 09, 2025, 10:06 AM UTC· resolved Aug 09, 2025, 10:17 AM UTC

1. investigating Aug 09, 2025, 10:06 AM UTC

   We acknowledged that the documentation website docs.mender.io is currently not serving traffic. We are investigating the issue

2. monitoring Aug 09, 2025, 10:08 AM UTC

   The issue has been identified on the web server serving traffic; as a temporary workaround, we forcefully scaled out the web server and we're observing the results.

3. resolved Aug 09, 2025, 10:17 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 23, 2025, 05:08 AM UTC· resolved Jun 23, 2025, 05:39 AM UTC

1. investigating Jun 23, 2025, 05:08 AM UTC

   We are investigating an issue regarding Redis cluster and the Device Auth Service which is in degraded state.

2. monitoring Jun 23, 2025, 05:19 AM UTC

   The issue has been identified: a new Redis pod was restarting because of OOMKill. More memory has been given to the Redis pool and now the services are up. We're monitoring the result.

3. resolved Jun 23, 2025, 05:39 AM UTC

   This incident has been resolved.

4. postmortem Jun 23, 2025, 08:29 AM UTC

   This morning, the operation team performed a planned Redis Cluster upgrade, starting at 04:40 UTC. Around 04:56 UTC, one of the Redis pod got killed because of Out of Memory issues, causing the Device Auth service to experience connection failure. To resolve this, the operation team increased the memory allocated to the Redis Cluster, starting at 05:05 UTC. The change was fully implemented by 05:14 UTC, and no more error log was seen from the Device Auth service, which was returned to normal operation.

Read the full incident report →

Started Mar 20, 2025, 09:25 AM UTC· resolved Mar 20, 2025, 11:02 AM UTC

1. investigating Mar 20, 2025, 09:25 AM UTC

   We are currently investigating an issue regarding the Inventory service on hosted Mender EU: we got alerted by an unusual amount of 500 error.

2. monitoring Mar 20, 2025, 09:52 AM UTC

   The issue has been identified and we forcefully scaled up the Inventory resources. Now all the metrics are fine but we're still continuing to monitor the issue.

3. resolved Mar 20, 2025, 11:02 AM UTC

   This incident has been resolved: a memory tuning has been applied to the Inventory service.

Read the full incident report →

Started Mar 06, 2025, 03:01 PM UTC· resolved Mar 31, 2025, 07:17 PM UTC

1. identified Mar 06, 2025, 03:01 PM UTC

   We identified an issue regarding the Server-side generation of Delta Artifacts feature; the generation starts and reports success but the silently fails in the background. We are working on a fix.

2. resolved Mar 31, 2025, 07:17 PM UTC

   A fix has been implemented and tested. This incident is closed.

Read the full incident report →