Is Hosted Mender down?

We are currently investigating this issue which has been reported by customers.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we're monitoring the results.

This incident has been resolved.

## Executive Summary We renew the signing key for our apt repositories every two years. The renewal on 2026-09-15 produced a key that Debian 13 \(trixie\) rejects. For about half a day, anyone installing Mender packages on Debian 13 got "`the repository is not signed`" and could not install anything. Existing devices were unaffected. Debian 12 and older were unaffected. No package was altered and nothing was exposed; this was a verification failure, not a security breach. ## What happened A signing key carries a signature saying "`this key is valid`". Renewing the key is supposed to extend that statement. The tool we use replaces it instead, so the renewed key effectively said "`valid as of today`" with no record of having been valid before. Debian 13 shipped a stricter signature checker. It asks whether the key was valid _at the moment the repository was signed_. Our repositories were last signed before the renewal, so the answer was no and it refused them. Debian 12 and older do not ask that question, which is why everything looked fine to us. ## Impact * Debian 13 users could not install or update Mender packages. ## Resolution We rebuilt the key so it carries both its old and its new validity statements. It is the same key with the same fingerprint, so nothing changed for existing devices, and it now satisfies both the old and the new checkers. Verified on Debian 12 and Debian 13, against every repository. **If you applied a workaround, re-fetch the key.** Some users unblocked themselves by fetching the key from `https://downloads.mender.io/repos/device-components/gpg`. During the incident that URL served the old key, which expires on **2026-10-01**, so those installs will start failing again in October. That URL now serves the corrected key, so fetch it once more: ``` curl -fsSL https://downloads.mender.io/repos/device-components/gpg \ | sudo gpg --dearmor -o /usr/share/keyrings/mender.gpg` ``` The expiry should now read 2028-09-14: ``` gpg --show-keys /usr/share/keyrings/mender.gpg ``` Adjust the output path if you installed the key somewhere else; the official docs place it at `/etc/apt/trusted.gpg.d/mender.asc`. No change to your `sources.list` is needed, and devices already running Mender need no action. ## Why we did not catch it 1. We have an internal renewal and verification runbook which doesn't include Debian 13, so we didn't catch it. 2. The repositories kept serving. Only clients on a newer distribution saw a problem, and we had no monitoring for that. ## Actions to avoid this issue the next time * We updated the internal runbook to also include Debian 13 and a note to check for future distributions. * We added a verification pipeline to the renewal process. ## Apology We are sorry. This was our mistake, and it blocked people from installing our software on a current, supported version of Debian. We have updated our internal renewal process to test against current Debian releases, and added a verification pipeline so this is checked automatically the next time. We did not find this ourselves. A customer did, and reported it. We are grateful for that report, which was precise enough that we could reproduce the problem within minutes, but catching this should not have been their job.

The latest release to hosted Mender on 2026-08-26T12:09Z introduced more optimized indexing of device inventory which had an unintended side effect of denying devices with array-type inventory data.

We have rolled back the affected service (inventory) to restore functionality for the affected devices. We are continuing to monitor the situation.

This incident has been resolved.

An upstream provider notified that they are experiencing elevated error accessing API on EU region. It's possible that the Mender devices are unable to pull artifacts and deployments might fail. We're monitoring the upstream incident status.

The upstream provider reported their incident is resolved.

We are currently investigating the root cause of the issue.

We found a possible root cause with the new intermediate certificate of https://eu.hosted.mender.io with a different size. A tested workaround is to increase the mbedTLS heap size in your application's prj.conf: CONFIG_MBEDTLS_HEAP_SIZE=98304.

This incident has been resolved by tuning the mbedTLS heap size.

## **Executive Summary** An issue was identified where microcontroller \(MCU\) devices, specifically those running Zephyr OS and `mender-mcu` on constrained hardware like the ESP32-S3, lost connection to the hosted Mender EU server \(`eu.hosted.mender.io`\). Specifically, devices failed during the initial TLS handshake with error `-0x2700`, preventing Over-The-Air \(OTA\) updates and inventory reporting. The investigation confirmed that the root cause was **insufficient client-side memory \(mbedTLS heap\)** when processing the server's RSA-4096 certificate chain, rather than a certificate validity or time-synchronization issue. To recover devices already deployed in the field without requiring a physical intervention, the engineering team deployed a **server-side mitigation by switching the production TLS certificate authority to Google Trust Services \(GTS Root R4\)**. This restored connectivity for affected field devices. ## **Impact** Customers using MCU devices targeting the EU instance were not able to perform updates from the 9th to the 15th of July. ## **Technical Root Cause** The hosted Mender EU endpoint utilized a cross-signed Let’s Encrypt certificate hierarchy ending in an RSA-4096 leaf and root certificate. From the 4th of June, letsencrypt \(EU\) started using a different chain from the same root CA, with one more intermediate. _Jan – early Jun 2026 │ leaf → R12/R13 → ISRG Root X1 --> 1 intermediate_ _Jun 4 2026 → now     │ leaf → YR1 → ISRG Root YR → ISRG Root X1 --> 2 intermediates_ This change landed at the beginning of July to hosted Mender EU, after automated cert-bot renewal, and then the mender-mcu clients lost connection. During the TLS handshake, mbedTLS requires continuous heap memory to simultaneously hold: * The TLS input and output content buffers \(`SSL_MAX_CONTENT_LEN`\). * The parsed peer certificate chain \(RSA-4096\). * The parsed trust anchors. * The temporary mathematical buffers required to execute an RSA-4096 signature verification. On constrained MCU devices, the default mbedTLS heap configuration \(`CONFIG_MBEDTLS_HEAP_SIZE=40960`, or ~40 KB\) was exceeded. Our tTesting revealed that processing this specific RSA chain required a minimum heap threshold of ~41.5 KB \(42496 bytes\). When the memory allocation failed internally during signature verification, mbedTLS surfaced the out-of-memory exception as a general certificate trust failure \(flag 0x08: `BADCERT_NOT_TRUSTED`\). A mitigation was published, to increase the `CONFIG_MBEDTLS_HEAP_SIZE`, but of course this can’t be applied to devices which lost the connection to the server. ## **Resolution & Recovery** ### **Immediate Field Recovery \(Server-Side\)** Because deployed devices with broken OTA connectivity cannot receive a firmware update to increase their local heap size, engineering deployed a server-side resolution: * Switched the production certificate chain for [eu.hosted.mender.io](http://eu.hosted.mender.io) from Let's Encrypt to **GTS Root R4 \(Google Trust Services\)**. * The GTS Root R4 chain is already bundled within the existing MCU client trust stores and requires significantly lower memory overhead during verification, allowing already-deployed devices to successfully complete the handshake and re-establish OTA connectivity without a client-side patch. * A previous attempt to force a lighter CA Chain with ECDSA root CA failed; although the required heap was less than the new RSA root, it was still heavier than the old RSA root, so our internal validation failed. ### **Long term solutions** * We are implementing an end-to-end failover strategy covering both TLS infrastructure and domain name \(DNS\) routing. * We implemented periodic tests to emulate a MCU device connecting to both US and EU instances to early catch this kind of issues. ‌ We sincerely apologize for the disruption this caused to your fleets, this was a serious failure, and we understand the operational burden it placed on you. We are committed to the improvements described above and to ensuring that changes in upstream certificate infrastructure never again silently break connectivity for constrained devices.

We are currently investigating this issue.

The issue has been identified: there was an issue with the Redis and NATS deployments. A fix has been implemented and we're restarting the impaired services.

The affected services have been restarted and we're monitoring the results.

This incident has been resolved.

# Brief service disruption on Hosted Mender \(EU\) On June 24, between 12:04 and 12:09 UTC, some customers on our EU region experienced a short interruption. During this five minute window, devices may have had trouble connecting, and parts of the web interface and API may have returned errors. ## What happened While cleaning up a configuration issue, we removed an internal duplicated definition \(the same set of kubernetes manifests rendered twice from Kustomize\). Our automation interpreted this as a request to remove and reinitialize two background components that handle messaging and caching \(NATS and Redis\). They were rebuilt automatically, but for about five minutes they were unavailable, which caused the disruption. ## What we are doing about it The automation completed the reconciliation loop and redeployed the background components automatically. We verified that normal operation was restored and confirmed all services are healthy. We are changing how we make this type of change so that components are never removed before their replacement is ready. We apologize for the disruption.