Opti9 incident

Intermittent Backup and Replication Job Issues - Calgary

Minor Resolved View vendor source →

Opti9 experienced a minor incident on December 29, 2025 affecting Veeam Cloud Connect Backup - hostedbizz-west2 and Veeam Cloud Connect Replication - hostedbizz-west2 and 1 more component, lasting 1h 43m. The incident has been resolved; the full update timeline is below.

Started
Dec 29, 2025, 03:08 PM UTC
Resolved
Dec 29, 2025, 04:51 PM UTC
Duration
1h 43m
Detected by Pingoru
Dec 29, 2025, 03:08 PM UTC

Affected components

Veeam Cloud Connect Backup - hostedbizz-west2Veeam Cloud Connect Replication - hostedbizz-west2Veeam Cloud Connect Backup - yyc01-bVeeam Cloud Connect Replication - yyc01-bVeeam Backup for Cloud Director - YYCVeeam Service Provider Console - console-yyc

Update timeline

  1. investigating Dec 29, 2025, 03:08 PM UTC

    We are currently experiencing intermittent backup and replication job issues that may fail and restore and failover operations may not be available. You may see error messages that you are unable to connect to the service provider.

  2. resolved Dec 29, 2025, 04:51 PM UTC

    We have resolved the intermittent issues impacting backup and replication jobs in our Calgary region. If you continue to observe failures or issues with your backup/replication jobs, please reach out to our technical support team.

  3. postmortem Jan 05, 2026, 03:40 PM UTC

    ## Incident Summary On Saturday, December 27th at approximately 1:00AM Eastern time, a VMware ESXi host running several servers supporting Veeam Cloud Connect crashed. These servers were subsequently HA \(high availability\) restarted on another host in the cluster. Several Veeam Cloud Connect Gateways lost WAN connectivity as a result of the restart. This issue was not immediately apparent, due to the LAN connectivity for these gateways being functional. The incident was fully resolved on Monday, December 29th at 10:15AM Eastern Time after repairing the WAN connectivity to the offline gateways and restarting the Veeam Cloud Connect service. ## Impact of Incident All jobs that were running at the time of the incident failed, due to the Veeam Cloud Connect server being impacted. After the restart, a subset of client jobs experienced failures due to being assigned an offline Veeam Cloud Connect gateway. ## Follow Up Actions Opti9 are reviewing the incident carefully, and making adjustments to monitoring and processes as required to prevent future incidents. At this time, we have made the following adjustments: · Expanded our health check procedures to include manual checks and verification of the Veeam Cloud Connect gateways · Added additional monitoring items, specific to the Veeam Cloud Connect gateway component to ensure they are functional at all times