Replicated Outage History

We're observing a higher than expected error rate for requests handled by an upstream networking provider. This may result in a subset of requests being met with high latency and 5xx errors. We are closely monitoring an ongoing incident with the upstream provider at this time for updates and resolution.

We are continuing to monitor updates provided by our upstream networking provider. The immediate impact that we are observing has been significantly reduced in the last 30 minutes. However, the issue is not fully mitigated or resolved in the upstream provider at this time.

A fix has been implemented and we are monitoring the results.

We're continuing to monitor upstream notices from Redhat regarding the quay.io incident. Latest messaging is that Pulls are operational.

There are possible delays when provisioning OpenShift on CMX. This relates to an ongoing incident with an upstream dependency. We are closely watching this and will update as conditions improve. Thank you for your patience.

https://docs.replicated.com is currently impacted by an incident with an upstream provider. We are closely monitoring this issue and will provide updates here. Thank you for your patience in this matter.

The issue looks to be resolved by azure and we're monitoring the issue.

CMX is impacted by an upstream incident for Azure AKS. We apologize for the inconvenience and are monitoring for improvements.

Internal end to end testing is showing successful operations with VMs and Clusters on CMX

CMX operations appear to be working as expected. We are continuing to monitor this situation and the upstream provider.

We are seeing some recovery from the ongoing Cloudflare incident. We are continuing to monitor.

We are continuing to monitor for any further issues.

Internal testing and evaluation of the Replicated platform components shows are are operational. Cloudflare is additionally noting that they believe the incident is resolved, though they are continuing to monitor. Should issues arise again we will keep our customers informed of conditions.

We are observing healthy responses from registry.replicated.com at this time. We have identified that error responses observed earlier related to unauthenticated pulls with specific versions of Helm. We are iterating on improvements to mitigate the cause of these problems before rolling back forward.

A fix has been rolled out, and we are monitoring the results. Vendors may need to refresh the webpage.

We have identified that provisioning new EKS clusters on CMX has a high failure rate. This appears correlate with an ongoing incident impacting multiple services at AWS. We are monitoring this situation closely.

We are beginning to see recovery for EKS on Compatibility Matrix. We will continue to monitor this and update as conditions change.

We've issued a rollback for the affected platform component and are monitoring for full recovery.

We are seeing network performance improvements now, but are still monitoring this issue.

We're experiencing significant network performance issues from customers and workloads in EU regions when communicating with Replicated platform.

We are continuing to monitor for any further issues.

We are beginning to observe improved performance for downloads with workloads in the EU at this time. kURL and Embedded Cluster installations on CMX appear to be completing successfully.

Creating GKE clusters through Compatibility Matrix appears to be impacted by an ongoing service disruption with GCP for GCE VMs in us-central. We are closely monitoring this incident with Google

After further review of the increase in HTTP 5xx errors, we've identified that these failed requests are generally coming through an upstream provider in a particular point of presence. We are monitoring ongoing issues with our upstream provider and will update here as conditions change.

Services appear to be recovering. We will continue to monitor the issue.