Umbrellar Outage History

Due to a breaking change today in the upstream Bootstrap CDN resources, the Umbrellar Portal is currently experiencing issues with UI rendering and layouts. This has been fixed by the Dev Team and the new release is currently working its way through testing before being released into Production. Apologies for the inconvenience.

The problem has been traced to a malfunctioning background sync task. This has been rectified to prevent any further inconsistencies, however we have confirmed some subscriptions have been duplicated and/or suspended incorrectly. We are currently working to identify all affected subscriptions to determine required remedial actions.

We have identified the root cause as being related to one of our authentication providers currently experiencing an outage. More information can be found here - https://status.auth0.com/incidents/ls8z6099myw0. We will advise on further updates as they happen. Thanks.

Failure of one of the dual power feeds in the datacentre facility caused some core networking equipment to restart while secondary equipment did not seamlessly pick up the remaining load. Our engineers are currently manually recovering services.

We have identified a failed node in our dual-site cluster based in our Christchurch datacentre. Our engineers are working with the hardware vendor to restore service.Customers may experience higher than normal disk latency while the node remains in a failed state.We will provide further updates as we receive information from our vendor.

We are currently experiencing an issue with our upstream telephony provider for our Service Desk, resulting in an outage to our telephone support channel. We are working with our provider to resolve things as soon as possible. In the interim, if you need to contact us please email us or use our live chat feature. Thank you.

New Umbrellar registrations are currently unavailable while we work to fix an identified issue. Registrations will be back online shortly, in the mean time please contact our Success Team on 0800 344 493. Apologies for the inconvenience.

Our engineers are on site with replacement parts and working to restore power and service as soon as possible. Apologies for the inconvenience.

Multiple power supply failures have tripped multiple circuit breakers causing 3/4 of the Umbrellar Cloud compute nodes to go offline. This is impacting all services delivered by Umbrellar Cloud. Engineers are currently on route to the datacentre facility to recover service.

Service has been partially restored. The Umbrellar Cloud Portal will now log you in and successfully load, but you may still experience poor performance or certain blades and views not loading. We are working hard to fully restore service. Thanks for your patience.

The Umbrellar Cloud Portal is currently unavailable. When you login you will see the loading animation indefinitely and the Portal does not actually load successfully. We have identified the issue and urgently working with our vendor to resolve. There is no impact to running resources on the platform. Please contact our support team if you are experiencing any other problems. Apologies for the inconvenience.

We have identified an issue with logins and authentication on the Umbrellar Cloud platform. Initial testing shows this is currently only affecting our administration logins, customer logins and resources are still up and running. We are working to resolve this as soon as we can. In the meantime, if you encounter any issues please contact our support team. Thank you.

Our team have isolated the issue and corrected the time on all internal NTP servers. Further investigations are ongoing to identify the root cause and to implement a permanent fix.

We have identified the cause of the outages to an internal DNS problem, and we have also confirmed impact to running workloads. We will provide further updates as soon as we have them. Apologies for the inconvenience.

While we have resolved the observed DNS issue within the Umbrellar Cloud platform, we have also identified an underlying storage issue which looks to be the original cause. While the Management Portal & API are now back online, the experience and resource deployments should be considered degraded until further notice. You will likely encounter errors if trying to deploy new resources. We are working hard to resolve things as fast as possible. Thank you for your patience.

The ability to link an existing Azure AD Tenant with Umbrellar, for both Customers and Resellers, is currently unavailable. We have traced the cause to an upstream provider issue, we have raised it with them as a critical priority and are working with them to resolve as soon as possible. Apologies for the inconvenience.

Identified We discovered an issue when user can't raise a support case through the Umbrellar Cloud Portal - customers can raise a case but they get an "Invalid Page" after submission.

We have identified an issue with giving new users permissions to access Umbrellar Cloud Portal, we are investigating the matter, All the users who already access to Umbrellar Cloud Portal portal are NOT affected. Sorry for the inconvenience.

We discovered additional related issue when user can't raise a support case through the Umbrellar Cloud Portal - customers can raise a case but they get an "Invalid Page" after submission.

We have identified an issue for customers who have enabled multi-factor authentication for guest users not being able to access Umbrellar Cloud Portal. We are going to implement new authentification scenario on our Cloud Portal within 7 days. Those customers who have multi-factor authentication for guest users enabled will not be able to access Umbrellar Cloud Portal however those customers who do not have this option turned on will be able to access portal without any problems. Sorry for the inconvenience.

We have been facing an issue when customers can't create new and modify existing subscriptions through Umbrellar Cloud Portal. We identified an issue and working on solution. Sorry for inconvinience

Disk write performance for a small number of Virtual Servers will be degraded due to a hardware failure. This hardware failure has now been resolved however, the backup jobs must be run to ensure backups are successful. This will be a temporary degradation while the backup is running for the relevant Virtual Server

We want to inform you that we are currently aware of an issue affecting Azure Front Door, which may be impacting your service availability or performance. Microsoft has acknowledged the incident and is actively working on a resolution. You can follow their updates directly via https://azure.status.microsoft/en-us/status. We are closely monitoring the situation and will keep you informed as we learn more. Your experience is important to us, and we appreciate your patience and understanding.

At this stage, Microsoft anticipate full mitigation within the next four hours as they continue to recover nodes. This means we expect recovery to happen by 12:20PM on 30 October 2025. We will provide another update on our progress within two hours, or sooner if warranted

Microsoft has deployed its last known good configuration and recovery is progressing. They are actively restoring nodes and rerouting traffic through healthy systems. While improvements are being seen, some requests may still hit unhealthy nodes, causing intermittent failures or reduced availability. The recovery process involves reloading configurations and rebalancing traffic across a large fleet. This is being done gradually to maintain stability and avoid overload as dependent services come back online. Microsoft expects full mitigation by 12:40PM NZDT on 30 October 2025, and will communicate if recovery completes sooner. Customer configuration changes remain temporarily blocked by Microsoft to prevent interference with recovery efforts. They will notify when this restriction is lifted.

We are continuing to work on a fix for this issue.