ThinkOn Outage History
ThinkOn is up right nowPingoru has recorded 53 incidents for ThinkOn: 31 in the last 12 months and 6 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 5 | 69 of 720 (10%) |
| May 2026 | 2 | 80 of 744 (11%) |
| June 2026 | 4 | 13 of 720 (2%) |
| July 2026 | 2 | 336 of 744 (45%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 2 | 9 of 720 (1%) |
| October 2026 | 3 | 100 of 576 (17%) |
Incidents, newest first
Critical November 29, 2025 Object Storage Inaccessible 2d MEL3 Object Storage
Started Nov 29, 2025, 04:45 AM UTC· resolved Dec 01, 2025, 05:34 AM UTC
- investigating Dec 01, 2025, 12:28 AM UTC
Object Storage is currently inaccessible.
- investigating Dec 01, 2025, 12:55 AM UTC
We are continuing to investigate this issue.
- resolved Dec 01, 2025, 05:34 AM UTC
Issue has been resolved. All the OBJECT STORAGE in MEL3 is accessible.
Minor November 24, 2025 [MEL3] Data Protect Portal Inaccessible 45m MEL3 Veeam M365
Started Nov 24, 2025, 04:43 AM UTC· resolved Nov 24, 2025, 05:28 AM UTC
- investigating Nov 24, 2025, 04:43 AM UTC
We are currently experiencing an issue where the Data Protect Portal is inaccessible. Our engineering team is actively investigating the root cause and working to restore access as quickly as possible.
- resolved Nov 24, 2025, 05:28 AM UTC
This incident has been resolved.
Major November 5, 2025 [OTT2/OTT2E] Virtual Machine Creation Failure 1h 24m OTT2 Virtual Compute
Started Nov 05, 2025, 02:26 PM UTC· resolved Nov 05, 2025, 03:51 PM UTC
- identified Nov 05, 2025, 02:26 PM UTC
We’ve identified an issue in the underlying infrastructure that is currently causing Virtual Machine creation to fail. Our team is actively working on a fix, and we expect the issue to be resolved shortly. We will provide an update within the next hour.
- resolved Nov 05, 2025, 03:51 PM UTC
This incident has been resolved.
Notice October 30, 2025 [OTT 1/2/3/4] - Facility Alert: Electrical Smell Investigation in Progress (No Service Impact) October 30th, 2025 1d 8h OTT2 Object Storage, OTT1 Rubrik, OTT3 Object Storage, OTT2 Virtual Compute and 4 more
Started Oct 30, 2025, 04:30 PM UTC· resolved Nov 01, 2025, 12:41 AM UTC
- investigating Oct 30, 2025, 04:30 PM UTC
We are currently Investigating a Burning Smell coming from our Electrical Rooms. Contractors are currently onsite and are investigating this issue. We have Transferred the Critical Loads to Generator Power. Electricians are being dispatched to review the issue. NO customer equipment or services are impacted and our team will monitor the activity closely.
- identified Oct 31, 2025, 04:11 PM UTC
**UPDATE** The faulty breaker has been replaced successfully. Site loads have been transferred back to hydro power. There was no impact to customer equipment or services during the process.
- resolved Nov 01, 2025, 12:41 AM UTC
Normal operations have been fully restored.
Notice October 29, 2025 NUBAV - Multiple Services Impacted 6h 3m Compass, NUBAV Object Storage, NUBAV Virtual Compute, NUBAV Zerto Virtual Replication
Started Oct 29, 2025, 06:55 PM UTC· resolved Oct 30, 2025, 12:59 AM UTC
- investigating Oct 29, 2025, 06:55 PM UTC
We are currently working with a vendor to resolve this issue.
- investigating Oct 29, 2025, 06:57 PM UTC
We are currently working with a vendor to resolve this issue.
- identified Oct 29, 2025, 08:06 PM UTC
Update: We are actively collaborating with our vendor to resolve the ongoing issue. The following services are currently impacted: - Virtual Compute & vCloud - Zerto Virtual Replication - Object Storage - Compass Portal We appreciate your patience and will provide further updates as progress is made.
- identified Oct 29, 2025, 08:42 PM UTC
We are actively collaborating with our vendor to resolve the ongoing issue. The following services are currently impacted: - Virtual Compute & vCloud - Zerto Virtual Replication - Object Storage - Compass Portal We appreciate your patience and will provide further updates as progress is made.
- identified Oct 29, 2025, 10:20 PM UTC
We are currently in the process of restoring object storage, vCloud, and other affected services. Our teams are actively working to bring systems back online as quickly and safely as possible. Next Update: Scheduled in 90 minutes Thank you for your continued patience. We’ll share further details as progress continues.
- identified Oct 30, 2025, 12:06 AM UTC
- Storage for dedicated blades has been successfully restored. However, some dedicated blades may require a manual reboot to fully reconnect to storage services. - All affected tenant VMs are now back online and operational. - Our teams are continuing work to restore vCloud access, and we will provide updates as progress continues. Thank you for your patience as we work toward full service recovery.
- resolved Oct 30, 2025, 12:59 AM UTC
We’re pleased to report that all systems are now back online, including storage, vCloud, and all affected virtual machines (VMs) and physical servers. What You Should Know: -Some virtual machines and dedicated blades may require a manual reboot to reconnect fully to storage. -If you experience any issues accessing your systems, please contact the Service Desk. We sincerely apologize for the inconvenience and appreciate your patience throughout the recovery process.
Notice September 5, 2025 [CYG3/CGY5] Network Outage 6h 57m CGY3 Internet Connectivity, CGY5 Virtual Compute, CGY3 Virtual Compute, CGY3 Object Storage and 3 more
Started Sep 05, 2025, 01:48 PM UTC· resolved Sep 05, 2025, 08:45 PM UTC
- investigating Sep 05, 2025, 01:48 PM UTC
We are currently experiencing a network outage at our CGY3 & CGY5 site. Our ThinkOn engineers are working with the vendor to find a resolution for this issue.
- investigating Sep 05, 2025, 03:08 PM UTC
Thinkon Team is continuing to troubleshoot with the Vendor.
- investigating Sep 05, 2025, 08:37 PM UTC
Thinkon Team continues troubleshooting the issue with the Vendor
- resolved Sep 05, 2025, 08:45 PM UTC
ThinkOn engineer worked with the vendor to resolve the issue.
Notice August 22, 2025 CMB1 - Network connectivity issue 1d 1h CMB1 Internet Connectivity, CMB1 Virtual Compute, CMB1 Veeam Cloud Connect, CMB1 Zerto and 1 more
Started Aug 22, 2025, 07:01 PM UTC· resolved Aug 23, 2025, 08:20 PM UTC
- investigating Aug 22, 2025, 07:01 PM UTC
We are currently investigating the issue.
- identified Aug 22, 2025, 07:17 PM UTC
We are currently working with a vendor to resolve the issue.
- identified Aug 22, 2025, 09:00 PM UTC
We continue working with a vendor to resolve the issue.
- identified Aug 22, 2025, 10:17 PM UTC
Our ThinkOn engineers are continuing to work with the vendor to resolve the issue.
- identified Aug 22, 2025, 10:35 PM UTC
Our vendor has dispatched field technicians to the site, and repair work is underway.
- identified Aug 23, 2025, 12:24 PM UTC
Fibre teams have successfully lashed the damaged fibers on both ends One cable has been fully spliced The fibres on the other end are currently being spliced
- monitoring Aug 23, 2025, 01:35 PM UTC
The fibre has been repaired and services have come back online. We are currently monitoring the network.
- resolved Aug 23, 2025, 08:20 PM UTC
This incident has been resolved.
Notice August 20, 2025 Power Disturbance at KLP1 DC 1h 28m KLP1 Internet Connectivity, KLP1 Virtual Compute
Started Aug 20, 2025, 03:33 PM UTC· resolved Aug 20, 2025, 05:01 PM UTC
- investigating Aug 20, 2025, 03:33 PM UTC
The DC team is currently investigating the power disturbance at the Data Centre.
- resolved Aug 20, 2025, 05:01 PM UTC
We would like to inform you that the recent power disturbance at our Kamloops Data Centre has been fully resolved. All systems are now stable and operational.
Critical July 23, 2025 CGY4 - vCloud portal is down 3h 22m CGY4 Virtual Compute
Started Jul 23, 2025, 06:15 PM UTC· resolved Jul 23, 2025, 09:38 PM UTC
- investigating Jul 23, 2025, 06:15 PM UTC
We are currently investigating this issue.
- investigating Jul 23, 2025, 08:14 PM UTC
We are continue to investigate the issue.
- resolved Jul 23, 2025, 09:38 PM UTC
This incident has been resolved.
Critical July 16, 2025 CMB1 - ObjectStorge service is down 38m CMB1 Object Storage
Started Jul 16, 2025, 02:30 PM UTC· resolved Jul 16, 2025, 03:08 PM UTC
- investigating Jul 16, 2025, 02:30 PM UTC
We are currently investigating this issue.
- resolved Jul 16, 2025, 03:08 PM UTC
This incident has been resolved.
Notice July 14, 2025 OTT2 / OTT2E - Key Management Server outage, affecting SOS service 3d OTT2 Object Storage
Started Jul 14, 2025, 03:52 PM UTC· resolved Jul 17, 2025, 04:21 PM UTC
- investigating Jul 14, 2025, 03:52 PM UTC
We still continue working with a vendor to resolve this issue.
- investigating Jul 14, 2025, 07:41 PM UTC
We are continuing to investigate this issue.
- identified Jul 14, 2025, 07:51 PM UTC
We are continuing to work with the vendor to restore SOS services.
- identified Jul 15, 2025, 04:12 PM UTC
ThinkOn Team is continuing to work with the vendor to restore SOS service.
- identified Jul 15, 2025, 05:32 PM UTC
ThinkOn Team is continuing to work with the vendor to restore SOS service.
- identified Jul 15, 2025, 05:32 PM UTC
We are continuing to work on a fix for this issue.
- resolved Jul 17, 2025, 04:21 PM UTC
This incident has been resolved.
Notice July 10, 2025 PHX2 HCPCS Service Unavailable 1h 45m PHX2 Object Storage
Started Jul 10, 2025, 07:17 PM UTC· resolved Jul 10, 2025, 09:02 PM UTC
- investigating Jul 10, 2025, 07:17 PM UTC
Our team is currently investigatng the issue.
- resolved Jul 10, 2025, 09:02 PM UTC
Our team has resolved the issue.
Notice July 9, 2025 OTT2 / OTT2E - Key Management Server outage, affecting SOS service and vCloud VM creation 15h 27m OTT2 Virtual Compute
Started Jul 09, 2025, 07:53 PM UTC· resolved Jul 10, 2025, 11:21 AM UTC
- investigating Jul 09, 2025, 07:53 PM UTC
We continue working with a vendor to resolve this issue.
- investigating Jul 09, 2025, 08:00 PM UTC
We are continuing to investigate this issue.
- resolved Jul 10, 2025, 11:21 AM UTC
This incident has been resolved.
Minor July 4, 2025 [MEL3] VMware Maintenance 49m MEL3 Virtual Compute, MEL3 Virtual Compute Backups, MEL3 Veeam Cloud Connect
Started Jul 04, 2025, 01:59 AM UTC· resolved Jul 04, 2025, 02:48 AM UTC
- identified Jul 04, 2025, 01:59 AM UTC
We’d like to inform you that the scheduled vCloud Director upgrade maintenance at our MEL3 Data Center is currently experiencing a delay. The work is now expected to extend approximately one hour beyond the original maintenance window. Our teams are actively working to complete the upgrade as swiftly and safely as possible. Thank you for your patience and understanding.
- resolved Jul 04, 2025, 02:48 AM UTC
This incident has been resolved.
Minor June 5, 2025 [OTT2] HCPS Object Storage performance degradation 5h 39m OTT2 Object Storage
Started Jun 05, 2025, 04:19 PM UTC· resolved Jun 05, 2025, 09:59 PM UTC
- investigating Jun 05, 2025, 04:19 PM UTC
The ThinkOn Team is currently working with the vendor to identify the issue.
- investigating Jun 05, 2025, 06:53 PM UTC
The ThinkOn Team is continuing to work with the vendor to address the issue.
- resolved Jun 05, 2025, 09:59 PM UTC
The ThinkOn Team has worked with the vendor to successfully resolve the issue.
Notice May 16, 2025 CMB1 Site is down 9h 56m CMB1 Internet Connectivity, CMB1 Virtual Compute, CMB1 Veeam Cloud Connect, CMB1 Zerto and 1 more
Started May 16, 2025, 05:26 PM UTC· resolved May 17, 2025, 03:22 AM UTC
- investigating May 16, 2025, 05:26 PM UTC
We are currently investigating this issue.
- investigating May 16, 2025, 07:35 PM UTC
We still continue to investigate this issue.
- investigating May 16, 2025, 09:46 PM UTC
We are continuing to investigate this issue.
- identified May 17, 2025, 12:11 AM UTC
Services in the CMB1 region are gradually coming back online. Our team continues to monitor the situation closely and is working to ensure full restoration as quickly and safely as possible.
- identified May 17, 2025, 12:19 AM UTC
We are continuing to work on a fix for this issue.
- monitoring May 17, 2025, 12:21 AM UTC
Services in the CMB1 region are gradually coming back online. Our team continues to monitor the situation closely and is working to ensure full restoration as quickly and safely as possible.
- resolved May 17, 2025, 03:22 AM UTC
This incident has been resolved.
Notice May 12, 2025 CMB1 VCloud is Down 7m CMB1 Internet Connectivity, CMB1 Virtual Compute
Started May 12, 2025, 08:13 PM UTC· resolved May 12, 2025, 08:20 PM UTC
- investigating May 12, 2025, 08:13 PM UTC
We are currently investigating this issue.
- resolved May 12, 2025, 08:20 PM UTC
This incident has been resolved.
Notice May 9, 2025 [MTL2] Network Connectivity 10h 26m MTL2 Internet Connectivity, MTL2 Object Storage, MTL2 Virtual Compute, MTL2 Veeam Cloud Connect and 1 more
Started May 09, 2025, 07:58 AM UTC· resolved May 09, 2025, 06:25 PM UTC
- investigating May 09, 2025, 07:58 AM UTC
We are aware of connectivity issues impacting our MTL2 site and are actively investigating.
- investigating May 09, 2025, 09:53 AM UTC
We are continuing to investigate this issue.
- investigating May 09, 2025, 10:24 AM UTC
The issue is resolved.
- investigating May 09, 2025, 10:24 AM UTC
The issue is resolved.
- resolved May 09, 2025, 06:25 PM UTC
This incident has been resolved.
Notice May 8, 2025 Service Outage: Probax Hive Object Storage 1d 16h
Started May 08, 2025, 02:07 AM UTC· resolved May 09, 2025, 06:26 PM UTC
- investigating May 08, 2025, 02:07 AM UTC
Please be advised that the Probax Hive Object Storage service is currently experiencing an outage. ThinkOn is actively working with the vendor to investigate and resolve the issue. Impact: This outage affects clients’ ability to perform backups to Object Storage provisioned through Hive. All other services remain unaffected at this time. There is currently no estimated time for resolution (ETA). We will continue to provide updates as new information becomes available. Thank you for your patience.
- investigating May 08, 2025, 02:08 AM UTC
Please be advised that the Probax Hive Object Storage service is currently experiencing an outage. ThinkOn is actively working with the vendor to investigate and resolve the issue. Impact: This outage affects clients’ ability to perform backups to Object Storage provisioned through Hive. All other services remain unaffected at this time. There is currently no estimated time for resolution (ETA). We will continue to provide updates as new information becomes available. Thank you for your patience.
- investigating May 08, 2025, 02:27 PM UTC
Please be advised that the Probax Hive Object Storage service is currently experiencing an outage. ThinkOn is actively working with the vendor to investigate and resolve the issue. Impact: This outage affects clients’ ability to perform backups to Object Storage provisioned through Hive. All other services remain unaffected at this time. There is currently no estimated time for resolution (ETA). We will continue to provide updates as new information becomes available. Thank you for your patience.
- resolved May 09, 2025, 06:26 PM UTC
This incident has been resolved.
Minor May 7, 2025 Service Interruption: Probax Hive Portal is down 12h 15m
Started May 07, 2025, 02:20 AM UTC· resolved May 07, 2025, 02:35 PM UTC
- investigating May 07, 2025, 02:20 AM UTC
Please be advised that the Probax Hive Portal is currently unavailable. ThinkOn is actively working with the vendor to resolve the issue. At this time, there is no estimated time for resolution (ETA). Important: This issue does not impact backups or any other services. We will provide updates as new information becomes available. Thank you for your patience.
- resolved May 07, 2025, 02:35 PM UTC
This incident has been resolved.
Major April 29, 2025 [OTT2] and [OTT2E] Partial operational outage 21h 24m OTT2 Virtual Compute
Started Apr 29, 2025, 08:33 PM UTC· resolved Apr 30, 2025, 05:57 PM UTC
- identified Apr 29, 2025, 08:33 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Apr 30, 2025, 05:57 PM UTC
This incident has been resolved.
Notice March 18, 2025 PHX2 HCPCS Degraded Service 1d 1h PHX2 Object Storage
Started Mar 18, 2025, 11:08 PM UTC· resolved Mar 20, 2025, 12:20 AM UTC
- investigating Mar 18, 2025, 11:08 PM UTC
We are facing a problem with one of our nodes at this time service may be degraded. If you experiencing a complete outage please submit a ticket to [email protected]
- monitoring Mar 19, 2025, 12:54 AM UTC
After a thorough investigation we can confirm this is not impacting your services, at this time our engineers will continue to monitor
- resolved Mar 20, 2025, 12:20 AM UTC
At this time we have not received any reports related to this incident and will be marking this as resolved. If any problems arise please submit a request to [email protected]
Critical January 30, 2025 [KLP1] Service Outage 40m KLP1 Internet Connectivity, KLP1 Virtual Compute
Started Jan 30, 2025, 05:33 AM UTC· resolved Jan 30, 2025, 06:14 AM UTC
- identified Jan 30, 2025, 05:33 AM UTC
We are currently experiencing an outage at KLP1. Our ThinkOn engineers have identified the issue and are working to resolve this issue.
- resolved Jan 30, 2025, 06:14 AM UTC
The issue is resolved, at this time if you have a active ticket open we will be marking it as resolved, if you are still having issues you may reply back to the ticket to reopen. For those who do not have a ticket and is still having problems, you can call/email the contact below.
Critical January 30, 2025 [CMH1] Service Outage 1h 35m CMH1 Internet Connectivity, CMH1 Virtual Compute, CMH1 Object Storage, CMH1 Zerto Virtual Replication and 1 more
Started Jan 30, 2025, 05:14 AM UTC· resolved Jan 30, 2025, 06:50 AM UTC
- identified Jan 30, 2025, 05:14 AM UTC
We are currently experiencing an outage at CMH1. Our ThinkOn engineers have identified the issue and are working to resolve this issue.
- resolved Jan 30, 2025, 06:50 AM UTC
This issue is now resolved. At this time if you have a active ticket open we will be marking it as resolved, if you are still having issues you may reply back to the ticket to reopen. For those who do not have a ticket and is still having problems, you can call/email the contact below.
Notice January 14, 2025 TOR10/OTT2 3h 16m TOR10 Virtual Compute, OTT2 Virtual Compute
Started Jan 14, 2025, 07:51 PM UTC· resolved Jan 14, 2025, 11:07 PM UTC
- identified Jan 14, 2025, 07:51 PM UTC
Problem with the TOR10/OTT2 Domain Controller, preventing changes being made to VMs in vcloud director.
- identified Jan 14, 2025, 08:19 PM UTC
We are continuing to work on a fix for this issue.
- resolved Jan 14, 2025, 11:07 PM UTC
This incident has been resolved.