ThinkOn had 50 outages in the last 2 years totaling 768h 25m of downtime — averaging 2.1 incidents per month.
There were 50 ThinkOn outages since November 18, 2024 totaling 768h 25m of downtime. Each is summarised below — incident details, duration, and resolution information.
KMS server crash, causing VM creation/manipulation operations to error out. ThinkOn engineers are working to resolve the issue
Increased KMS server disk size and restart. This incident is now resolved.
Please be advised that we are experiencing the following service interruption: MEL3 - VBO jobs are failing. Our team has identified the issue and is working on resolving the issue, in conjunction with escalated Veeam technical support.
We are continuing to work on a fix for this issue.
This incident has been resolved.
Our vendor had experienced a connectivity issue within the Data Centre and are working to resolve the issue.
The connection has returned, ThinkOn is monitoring the connection to ensure stability.
This incident has been resolved.
We are investigating an issue causing intermittent service interruptions. Customers may experience periods where the service is unavailable before connectivity is restored. Our teams are actively investigating and working to stabilize the service.
This incident has been resolved.
Dear valued customer: Please note our Ottawa data center is currently running on generator power due to a HydroOttawa power outage. There is no impact to customers services. We will provide an update once utility power is restored.
HydroOttawa power has been restored to the data center Generator has also stopped running. Site has successfully been transferred back to utility power.
CGY4 is currently experiencing network connectivity issues.
vCloud Access is Unavailable
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
This issue has been resolved. We are observing successful job completions.
We are currently investigating the issue. ThinkOn engineers are working with the vendor (veeam support) to resolve the issue.
We are continuing to investigate this issue.
We are continuing to investigate the issue. ThinkOn engineers are working with the vendor (Veeam support) to resolve the issue.
Our engineering team has successfully resolved the issue. If you notice failures after today please reach out to the ThinkOn Service Desk.
We are currently experiencing a service disruption affecting our Veeam M365 service hosted in the CGY4 data centre. Our engineering team is actively investigating the issue and working to restore services as quickly as possible. At this time, you may experience interruptions or delays when accessing or using the service. We understand the importance of this service to your operations and appreciate your patience while we work toward resolution. Thank you for your understanding."
We are continuing to investigate a service disruption affecting our Veeam M365 service in the CGY4 data center. Service interruptions or delays may persist. We appreciate your continued patience as we work toward resolution.
We are continuing to investigate a service disruption affecting our Veeam M365 service in the CGY4 data center. Service interruptions or delays may persist. We appreciate your continued patience as we work toward resolution.
We have isolated the issue and are currently working toward a solution. We’ll provide further updates as soon as we have more information.
We are continuing to work towards a resolution.
Our team has confirmed the issue has been resolved and systems are now operational.
We are currently investigating an issue affecting vCloud accessibility. At this time, we do not believe any workloads are impacted. Our team is working diligently to restore access as quickly as possible.
This incident has been resolved.
We are currently experiencing connectivity issues at our Ottawa Data Center. As a result, some customers may be experiencing service disruptions at this time. Our technical teams are actively investigating and working to restore full service as quickly as possible. We will continue to provide updates as more information becomes available. We appreciate your patience and understanding.
This incident has been resolved.
We are currently experiencing an issue affecting access to our Critical Compute environment, which may impact connectivity to vCloud services. At this time, there is no indication that customer workloads are impacted. Our team is actively investigating and working to restore full functionality as quickly as possible. We understand the importance of these services and appreciate your patience while we resolve the issue. Further updates will be provided as more information becomes available. If you have any urgent concerns, please reach out to our support team.
This incident has been resolved.
We are currently experiencing an issue accessing the vCloud portal. Our team is actively investigating the problem.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently experiencing a network issue impacting Zerto replication, Veeam backup operations, and overall system performance. Engineering teams are actively investigating and working to mitigate the issue.
We are still experiencing slowness with Zerto, VCC, VBO and we are still investigating. Engineering teams are actively investigating and working to mitigate the issue.
Performance has improved across affected services, including Zerto, VCC, VBO, and overall system operations. Customer reports and system metrics indicate services are stabilizing. ThinkOn engineers will continue to monitor closely to ensure stability is maintained. Updates will be provided if required.
This incident has been resolved
We are currently experiencing an issue accessing the Data Protect portal. Our team is actively investigating the problem.
The service disruption with the Data protect portal is now resolved.
Please be advised that we are currently troubleshooting a network situation: We are currently investigating a network issue at CGY4. While we troubleshoot and address the root cause, the connection to a few services may become intermittently, such as the vCloud portal and Zerto replications. We apologize for any inconvenience this may cause and appreciate your patience while we work to resolve the problem. Further updates will be provided as they become available.
This Incident has been resolved.
We are receiving multiple reports of service issues at the MTL2 site. Our team is actively working to identify the scope and root cause. Updates to follow.
We are continuing to investigate this issue.
We are working on resolving the issue.
Services have been restored.
We are currently aware of an issue that has caused network instability within our vCloud environments Our team believes they have resolved this issue - but we are monitoring to ensure stability and performance is as expected.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
Primary link connectivity came back. An investigation has begun as to why the fail over did not function as intended.
We are currently experiencing network connectivity issues that may impact service availability and performance for some customers. Our engineering and operations teams are actively troubleshooting the issue and working to restore normal service levels as quickly as possible. We will continue to provide updates as more information becomes available. Thank you for your patience while we work to resolve this.
The Network service degradation has been resolved.
Object Storage is currently inaccessible.
We are continuing to investigate this issue.
Issue has been resolved. All the OBJECT STORAGE in MEL3 is accessible.
We are currently experiencing an issue where the Data Protect Portal is inaccessible. Our engineering team is actively investigating the root cause and working to restore access as quickly as possible.
This incident has been resolved.
We’ve identified an issue in the underlying infrastructure that is currently causing Virtual Machine creation to fail. Our team is actively working on a fix, and we expect the issue to be resolved shortly. We will provide an update within the next hour.
This incident has been resolved.