Hosted Mender Outage History

Started Mar 20, 2025, 09:25 AM UTC· resolved Mar 20, 2025, 11:02 AM UTC

1. investigating Mar 20, 2025, 09:25 AM UTC

   We are currently investigating an issue regarding the Inventory service on hosted Mender EU: we got alerted by an unusual amount of 500 error.

2. monitoring Mar 20, 2025, 09:52 AM UTC

   The issue has been identified and we forcefully scaled up the Inventory resources. Now all the metrics are fine but we're still continuing to monitor the issue.

3. resolved Mar 20, 2025, 11:02 AM UTC

   This incident has been resolved: a memory tuning has been applied to the Inventory service.

Read the full incident report →

Started Mar 06, 2025, 03:01 PM UTC· resolved Mar 31, 2025, 07:17 PM UTC

1. identified Mar 06, 2025, 03:01 PM UTC

   We identified an issue regarding the Server-side generation of Delta Artifacts feature; the generation starts and reports success but the silently fails in the background. We are working on a fix.

2. resolved Mar 31, 2025, 07:17 PM UTC

   A fix has been implemented and tested. This incident is closed.

Read the full incident report →

Started Feb 25, 2025, 04:05 PM UTC· resolved Feb 26, 2025, 02:13 PM UTC

1. identified Feb 25, 2025, 04:05 PM UTC

   We are aware that some customers cannot update their credit card information from the Organization and Billing page. The issue has been identified and a fix is under deployment. This issue is affecting some customers on basic and professional plans.

2. monitoring Feb 26, 2025, 02:04 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 26, 2025, 02:13 PM UTC

   This incident has been solved: customers can now update billing details again.

Read the full incident report →

Started Feb 06, 2025, 08:39 AM UTC· resolved Feb 06, 2025, 10:33 AM UTC

1. investigating Feb 06, 2025, 08:39 AM UTC

   We are currently experiencing issues with the default Artifacts Storage provided by Cloudflare R2; we are investigating the issue. Deployment Service may be affected.

2. identified Feb 06, 2025, 08:51 AM UTC

   The issue has been identified: the default Artifacts Storage provider is experiencing an outage.

3. monitoring Feb 06, 2025, 09:43 AM UTC

   The upstream Storage provider implemented a fix and the storage seems available again. We see requests are now working. We'll continue to monitor the issue.

4. resolved Feb 06, 2025, 10:33 AM UTC

   The upstream provider declared their incident is resolved, so declaring incident as resolved on our end as well.

Read the full incident report →

Started Feb 03, 2025, 06:57 PM UTC· resolved Feb 04, 2025, 05:27 PM UTC

1. investigating Feb 03, 2025, 06:57 PM UTC

   We are aware that emails are not received from customers. We're currently investigating the issue

2. identified Feb 03, 2025, 07:16 PM UTC

   The issue has been identified. We're working on a possible solution.

3. identified Feb 03, 2025, 09:29 PM UTC

   We are continuing to work on a fix. We just introduced higher rate limits for devicemonitor, and requested a quota increase to our email provider, to cope a sudden burst of email request.

4. identified Feb 04, 2025, 10:43 AM UTC

   The fix has been applied: there is no new email sending burst at this time; unfortunately, the email provider quota is still exceeded and we're waiting for an increase, so emails are still not working.

5. monitoring Feb 04, 2025, 04:26 PM UTC

   We got our email quota increase request accepted and we're sending emails again. We're still monitoring the issue.

6. resolved Feb 04, 2025, 05:27 PM UTC

   This incident has been resolved; new emails are correctly sent.

7. postmortem Apr 04, 2025, 07:33 PM UTC

   **Abstract** Some hosted Mender customers were using the monitoring addon, but a bug in the client caused state flapping and a massive amount of emails to be generated and sent through the email provider. This resulted in hitting the rate limit, allowing hosted Mender to send 176,800 emails in about 3 hours before being rate limited for 24h from the email provider. This occurred for 4 days \(2 working days\) starting from the night of the 29th \(Wednesday\) before we acknowledged the issue on Monday evening the 3rd of February. This resulted in emails not being sent both from the monitoring add-on and from the password reset links, causing customers to not receive emails. After acknowledging the issue, we soon asked the email provider to raise the quota up to three times, and at the same time we got in contact with the hosted Mender customers which were using the monitoring add-on and asked them to temporarily mute the emails from the UI. As soon as the email quota was raised up, the email was flowing again. **Incident timeline** * 2025-01-29 16:00 UTC: The email provider reported an high email volume * 2025-01-29 19:00 UTC: The email provider quota was filled up and no emails from hosted Mender were sent anymore. * 2024-01-30 18:00 UTC: The email quota was cleaned up and hosted Mender started to send emails again. * 2025-01-30 21:00 UTC: The email provider quota was filled up and no emails from hosted Mender were sent anymore. * Same during the weekend: from 2025-01-31 18:00 UTC to 21:30 UTC, from 2025-02-01 19:00 UTC to 22:30 UTC and from 2025-02-02 19:00 UTC to 23:00 UTC emails have been sent. During all the other timeframes, the emails were not sent. * 2025-02-03 18:41 UTC: customers reported emails not working and the SRE team has been engaged * 2025-02-03 18:57 UTC: the incident has been opened * 2025-02-03 20:57 UTC: we increased the internal rate limit for deviceauth API requests, to slow down the emails flood. * 2025-02-03 21:50 UTC: we requested a service quota increase to Amazon SES * 2025-02-03 22:00 UTC: we requested the affected customers to mute the email monitor from the UI * 2025-02-04 15:19 UTC: the email provider increased the service quota: emails started flowing again. ‌ **Actions we have decided to take to avoid the same incident to happen again** * After a similar incident occurred in the past, we started developing an automated email synthetic test; we now finished this development and activated the new alert, to catch email problems in a timely manner. * We’ll split the SMTP service into two: one dedicated to the monitoring add-on, and another one for all the other important emails from hosted Mender

Read the full incident report →

Started Jan 13, 2025, 03:01 PM UTC· resolved Jan 13, 2025, 04:37 PM UTC

1. investigating Jan 13, 2025, 03:01 PM UTC

   We are experiencing scalability issue: new Kubernetes worker nodes are rolled out very slow. We're checking with the cloud provider.

2. monitoring Jan 13, 2025, 03:13 PM UTC

   Now the required load is matching the required number of Kubernetes worker nodes. We're still in contact with the cloud provider support to check the root cause. The incident is still open.

3. resolved Jan 13, 2025, 04:37 PM UTC

   The cloud provider support is still checking the issue. In the meantime we managed to increase the minimum number of Kubernetes worker node to prevent further autoscaling issue.

4. postmortem Jan 29, 2025, 09:15 AM UTC

   We discussed the incident with Azure support and decided to replace a problematic component \(an AKS Nodepool\). The new component is working fine and has no scalability issues, so we promoted it to production. No further actions are needed

Read the full incident report →

Started Jan 08, 2025, 10:06 AM UTC· resolved Jan 08, 2025, 02:07 PM UTC

1. investigating Jan 08, 2025, 10:06 AM UTC

   Today between 09:26 UTC and 09:28 UTC we got notifications about a MongoDB node failure on the primary. We are investigating the issue, that seems to be already solved.

2. identified Jan 08, 2025, 10:07 AM UTC

   We observed in the provider's log that it tried twice to roll it back, then the cluster gave up and elected a new primary. The cluster is self-healing and hosted Mender is operational again.

3. monitoring Jan 08, 2025, 10:07 AM UTC

   We're monitoring the incident and the metrics to check for possible issues.

4. resolved Jan 08, 2025, 02:07 PM UTC

   This incident has been resolved: the MongoDB cluster seems stable and no other issue has been reported.

Read the full incident report →

Started Dec 19, 2024, 10:15 AM UTC· resolved Dec 19, 2024, 01:41 PM UTC

1. investigating Dec 19, 2024, 10:15 AM UTC

   After the update from this morning, the hosted Mender artifacts are offered not anymore from the default storage, which is Cloudflare, but from the internal storage proxy, from which Cloudflare is not configured. This seems a misconfiguration and we're investigating the issue

2. monitoring Dec 19, 2024, 10:30 AM UTC

   A fix has been implemented and we're monitoring the results

3. resolved Dec 19, 2024, 01:41 PM UTC

   This incident has been resolved, the functionality is fully restored.

4. postmortem Dec 28, 2024, 02:35 PM UTC

   **Abstract** We upgraded hosted Mender with the Helm Chart v6.0.0, on the morning of the 19th of December, as planned; this new Helm chart includes the DEPLOYMENTS\_STORAGE\_PROXY\_URI variable already set, to facilitate open source users onboarding, In hosted Mender US, this variable was overridden for serving a single customer, so it was added with a default value. This caused an unexpected behavior that was to enable the storage proxy feature to every customer that has not the storage setting overridden, like most of the non enterprise customers. Luckily the storage proxy feature was not needed anymore, because the single customer using it had been migrated two weeks before, so we solved the situation by disabling the storage proxy feature entirely. **Incident timeline** * 2024-12-19 05:30 UTC: Mender Server v4.0.0-rc.4 and Helm Chart v6.0.0-rc.4 was applied to the hosted Mender US cluster with the storage proxy feature enabled * 2024-12-19 10:03 UTC: an operator was monitoring the status after the upgrade and found two support tickets from customers complaining about artifacts not getting downloaded. * 2024-12-19 10:15 UTC: this Statuspage incident was opened * 2024-12-10 10:30 UTC: we disabled the storage proxy feature from hosted Mender US and verified the download was successfully again * 2024-12-10 11:00 UTC: customers confirmed that the situation was restored again ‌ **Actions we have decided to take to avoid the same incident to happen again** The only customer that was using the storage proxy feature has already been migrated to a custom and supported solution, right to avoid this kind of issue. Additionally, we have to better monitor the artifacts download, and we’ll introduce more statistics, metrics and alerts.

Read the full incident report →

Started Dec 13, 2024, 08:09 AM UTC· resolved Dec 13, 2024, 12:44 PM UTC

1. investigating Dec 13, 2024, 08:09 AM UTC

   We are currently investigating the issue.

2. monitoring Dec 13, 2024, 08:23 AM UTC

   A fix has been implemented and we're monitoring the result

3. resolved Dec 13, 2024, 12:44 PM UTC

   This incident has been resolved.

4. postmortem Dec 19, 2024, 03:31 PM UTC

   **Abstract** The documentation website is hosted in a GKE cluster, which requires a Docker Hub user to pull private images to update the documentation pods. This Docker Hub user was supposed to be a new user, ready to be used, but until the 13th, still an old docker hub user was used. The issue started on the 11th when the old user’s password has been rotated for a security improvement, and the operator which performed the rotation didn't check that the user was still in use in the GKE cluster. After a few days the GKE cluster performed a node upgrade, and the pods were rescheduled, but could not pull the image because of the rotated Docker user password. The on-call team gets alerted when the documentation website was not working, and the operator created the new token for the right user this time, and the incident was solved. ‌ **Incident timeline** * 2024-12-11 - An operator, following a security ticket , updated the token for the a Docker hub user and changed the secret to an internal Staging cluster. The operator thought the secret was no longer used in the GKE Website cluster, and that there another user was used. * 2024-12-13 07:50 UTC - GKE performed a node rotation, causing the pods to be rescheduled * 2024-12-13 07:55 UTC - The docs website pods were using the old user’s secret, already rotated, so they didn’t have access anymore to the private repository and the pod was marked as ImagePullBackOff and was not starting * 2024-12-13 07:57 UTC - the monitoring system alerted the on-call operator * 2024-12-13 08:20 UTC - the operator created a new token for the right user and manually replaced the previous dockerconfigjson secret with the new one. The pods were running again * 2024-12-13 08:23 UTC - the monitoring system declared the website was up again ‌ **Actions we have decided to take to avoid the same incident to happen again** We have to refine the GKE cluster documentation, complete the migration to the new dedicated Docker hub user, and move the token to our internal secret renew automation tool.

Read the full incident report →

Started Nov 15, 2024, 11:39 AM UTC· resolved Nov 15, 2024, 02:04 PM UTC

1. investigating Nov 15, 2024, 11:39 AM UTC

   Today at 11:15 UTC, we observed a sudden spike in workload resources on the hosted Mender. This was accompanied by an increase in API errors. By 11:24 UTC, metrics returned to normal levels, and there have been no further indications of ongoing issues. The incident appears to have been transient and is no longer impacting the system's functionality. We are actively investigating the root cause to ensure stability and prevent recurrence.

2. identified Nov 15, 2024, 11:50 AM UTC

   The issue has been identified; the database scaled up automatically to address the increased load, so no further actions are needed.

3. monitoring Nov 15, 2024, 11:52 AM UTC

   We are monitoring the hosted Mender statistics and checking for new possible spikes.

4. resolved Nov 15, 2024, 02:04 PM UTC

   This incident has been resolved.

Read the full incident report →