Is CloudCone down?

We have identified an outage on our services, at this stage, some services should be facing network timeouts Our network engineers and sysadmins are investigating the issue at the moment. Further findings and updates will be updated on this status page.

The core issue is isolated to the upstream DWDM connections. The upstream providers are looking into the issue at the moment.

Services have been taken online via redundant measures. Network has been switched to GRE tunnelling, customers may experience degraded network performance, however it’ll keep traffic flowing until the provider resolves the core issue with the DWDM connections. We are also awaiting secondary DWDM uplink from our upstream providers to prevent such issues in the future. Currently the network is being closely monitored

Our team is currently investigating intermittent network timeouts and SSL/TLS connectivity issues affecting some services hosted in our Los Angeles (DC3) infrastructure following the recent datacenter migration. Affected customers may experience one or more of the following: Intermittent network timeouts. HTTPS/SSL connections failing or timing out during the TLS handshake. Inability to reach certain third-party websites, APIs, or package repositories. Software updates, and other outbound HTTPS requests intermittently failing. Slower than expected connectivity to specific network destinations. These issues do not affect all destinations or all customers and may be intermittent depending on the network path. Our networking team is actively investigating the underlying cause and implementing corrective measures to restore normal connectivity.

Update: Network upgrade scheduled to resolve connectivity issues Our network team has identified the cause of the intermittent connectivity issues affecting services in the Los Angeles data center. A network upgrade is scheduled to be implemented to address the underlying issue. Once completed, this change is expected to resolve the connectivity problems currently being experienced, including intermittent timeouts and connection failures. During this maintenance window, customers may continue to experience intermittent network-related issues until the changes are fully applied and validated. We will continue monitoring the network after the upgrade is completed and will provide further updates here. Thank you for your patience and understanding while we work to resolve this issue.

Our engineering team has completed the preparation and deployment of the required network hardware and supporting infrastructure for the planned network improvements. At this stage, we are awaiting a patch from our upstream provider, which is required before the remaining changes can be implemented. Once the patch has been applied, we will proceed with the final stages of the network upgrade and continue monitoring service stability. We will provide another update as soon as we have further progress to share. Thank you for your continued patience and understanding.

Our engineering team remains engaged with our upstream provider as we work through the final stages of the planned network upgrade. The required upstream patch is currently being applied, and we expect the remaining network changes to be completed shortly thereafter. Once the implementation is complete, we will continue monitoring service stability to ensure the connectivity issues have been resolved. We appreciate your continued patience and will provide another update once the work has been completed or if there are any significant developments.

Our network team is currently performing a hardware and upstream addition as part of the ongoing recovery efforts. During this swap, customers are experiencing network connectivity issues to services. This hardware replacement is the first of two planned stages required to restore the network to its original design and operating state. Once this stage is complete, network stability is expected to improve, with the remaining work to follow as scheduled.

The network improvements have now been largely completed, and we expect any remaining connectivity issues to be minimal throughout today. Our engineering team will continue monitoring the network closely, with the remaining work expected to be fully completed soon. Thank you again for your patience and understanding

Traffic has been transferred to Cisco ASR. IPv6 should be stable now, along with the TLS handshake issue that caused due to the lower MTU has been addressed. Currently the known issue related to IPv4 ranges experiencing short timeouts at ramdom intervals is being addresssed.

We have been alerted of an full network outage at the moment, which is being looked at.

The full network outage has been taken care of. Services should be accessible. The underlying Intermittent Network Timeouts issue still persists, and is being worked on.

Incident is now affecting a minor set of services, we are currently monitoring the network for further discoveries.

The network improvements have now been fully completed, and the incident has been resolved. Our engineering team has completed the network upgrades and verified the stability of the environment. The previously reported intermittent TLS handshake issues, IPv6 connectivity issues, and intermittent IPv4 timeout events have all been addressed. The network will continue to be monitored as part of our normal operations.

Our network is experiencing intermittent timeouts on the SC2 services. Our initial investigation shows that the problem appears to be isolated on the new DC (LAX) at the moment. Our engineers are working on a fix

There is a network outage happening at the moment. Our network engineers are looking into the issue at the moment

Services should be reachable now

We have identified an outage on our services, at this stage, some services should be facing network timeouts and degraded performance. Our network engineers and sysadmins are investigating the issue at the moment. Further findings and updates will be updated on this status page.

All affected hypervisors are back online.

We have identified RAID outage on our S160-V node in Los Angeles DC, Virtual instances on this specific node should see network timeouts. Our network engineers and sysadmins are investigating the issue at the moment. Further findings and updates will be updated on this status page.

The RAID 10 array could not be recovered despite recovery attempts. The node has now been brought back online to allow customers to proceed with restoring their data from backups after re-installing the VM. Customers are advised to begin the restoration process at their earliest convenience.