Amazee had 39 outages in the last 2 years totaling 3609h 23m of downtime — averaging 1.6 incidents per month.
There were 39 Amazee outages since September 10, 2024 totaling 3609h 23m of downtime. Each is summarised below — incident details, duration, and resolution information.
There is a delay in log processing and recent logs for events generated since June 1st may not be available. Real time logs can be access through the Lagoon CLI: https://docs.amazee.io/cloud/logging/#real-time-container-logs-via-lagoon-cli
This incident has been resolved.
We are currently experiencing an issue with cert-manager in affected clusters. The cert-manager controller pod is failing to start due to an image pull failure from quay.io (502 Bad Gateway), which is an external registry. Impact: New TLS certificates cannot be issued or renewed while cert-manager is unavailable. Existing certificates remain valid until their expiry. Services relying on automatic certificate provisioning may be affected if certificates expire during this window. https://status.redhat.com/
All clusters are able to pull cert-manager image, we are currently monitoring statuspage https://status.redhat.com/
This incident has been resolved.
We are aware of CVE-2026-31431 / "Copy Fail" and have completed an initial assessment. We have not identified evidence of customer impact, service compromise, or active exploitation at this stage. We are prioritizing Linux kernel upgrades and will provide further updates if our assessment changes or if maintenance becomes necessary.
Mitigations at the Linux kernel level have been rolled out across all regions.
We are currently investigating a high error rate on the Fastly management API. This impacts deployments of new domains that are routed through Fastly. Existing setups on Fastly or traffic routing are not impacted.
Fastly is investigating the API issues and updating their status page at: https://www.fastlystatus.com/incident/378511 Impact: Deployments with new domains that are routed through Fastly will continue to work, but the actual domain setup in Fastly will be delayed.
Fastly resolved the issue with their management API and we'll keep monitoring the situation.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
There was an issue with an EKS rollout during maintenance, we are actively addressing this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We have identified an issue causing Lagoon builds to fail and are actively working to resolve.
We have applied a fix for this issue and are monitoring deployments.
The incident has been resolved.
We are investigating increased error rates on the Lagoon API.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
In the latest release of Google Chrome (v145) the handling of HTTP 425 responses was changed. Should you see this affecting your sites, please reach out to our support and we'll look into applying mitigations for your specific use-case. Related status page from Fastly: https://www.fastlystatus.com/incident/378300#
This incident has been resolved.
Fastly has reported elevated errors affecting Fastly Applications. Existing certificates do not appear to be impacted; however, some services are experiencing issues when adding new domains. We are monitoring the Fastly status page: https://www.fastlystatus.com/
This incident has been resolved.
We're investigating intermittent SSH access to cloud clusters
We have identified and mitigated the cause for the SSH access issues, and are currently monitoring the portal.
This incident has been resolved.
We found an HTTP connection errors issue. The underlying issue was resolved, and we are actively monitoring the status.
This incident has been resolved.
We have seen an increase in failed deployments and are currently investigating the cause for this.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
The logging infrastructure is rebalancing due to the new month starting which causes some requests to fail.
There is a backlog of logs to be ingested into the system. While this is happening in the background, some recent logs might not be visible in the logging system right away.
The backlog has been processed and the logging system is fully available again.
New workloads on CH4 are unable to start. Deployments are expected to fail currently.
We have escalated the issue to the infrastructure provider and are working with them to resolve the issue.
We applied a fix which restored all functionality around the starting of workloads and doing deployments.
This incident has been resolved.
We are currently investigating this issue.
There was an unexpected rise in load on the logging system. We are now stabilizing the different components.
This incident has been resolved.
We have received multiple reports of connection interruptions when using the production databases on CH4. We are investigating the cause of these interruptions.
Since these issues are related to elevated resource usage we are doing an emergency resource increase. We expect a short interruption for existing connection which will help to prevent further disruptions.
This incident has been resolved.
We have identified a problem with logs ingestion and access. Engineers are investigating. We will provide an update within the next 90 minutes.
The logging system is working through a backlog of ingested logs that need to be allocated and indexed. While the backlog is being worked on, more data will become available over time. As the indexing is a resource intensive task, some requests can occasionally result in an error.
Due to the large backlog of logs, some log messages from 2025-11-01 to 2025-11-03 could not be loaded into the new logging system and had to be dropped. These logs are however still available in the old logging system at https://kibana-legacy.amazeeio.cloud.
Due to an upstream issue within AWS some workloads could not be scaled or deployed. The issue also affects our support system which is only partially available.
Some workloads in us2.lagoon are blocked from starting up due to the upstream issue. Other regions are no longer affected by this.
We are no longer seeing workloads affected by this issue.
The Lagoon UI and API are currently experiencing an issue, we're investigating.
We experienced an issue with image pulls during maintenance that resulted in the prolonged outage. This has been resolved now and we will continue to monitor.
This incident has been resolved.
Some workloads on UK3 are not responding as expected due to high load across a part of the compute infrastructure.
The situation has been stabilized and we continue to monitor the system closely
This incident has been resolved.
There is a delay in log processing and recent logs are not available in the logging system. Real time logs can be access through the Lagoon CLI: https://docs.amazee.io/cloud/logging/#real-time-container-logs-via-lagoon-cli
We are rolling out a potential solution which can cause connectivity issues with the Lagoon API.
While the logs are being processed the logging system can be unavailable at times.
The processing of some older logs is still ongoing.
The historical logs have been processed and are available in the logging stack.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
During the rollout of enabling DNS-01 challenges in our infrastructure, the image registry was unavailable due to temporary DNS misconfiguration.