Service disruption on all CVaaS regions

Update timeline

1. investigating Jun 12, 2025, 06:46 PM UTC

   We are currently investigating issue with our cloud provider. Some features may not be operational at this time. We'll provide update as soon as possible.

2. investigating Jun 12, 2025, 07:04 PM UTC

   Network Provisioning stack that involves pushing new configuration changes is down at this time. Viewing historical change control or other network changes is expected to be functional.

3. identified Jun 12, 2025, 08:21 PM UTC

   All non-US (cv-prod-us-central1-a, cv-prod-us-central1-b, cv-prod-us-central1-c) service regions and their functionality is expected to be operational. Please feel free to resume your usage of CVaaS on the following service regions: - cv-prod-india-1 - cv-prod-uk-1 - cv-prod-ausoutheast-1 - cv-prod-apnortheast-1 - cv-prod-euwest-2 - cv-prod-na-northeast1-b

4. monitoring Jun 12, 2025, 08:51 PM UTC

   All service regions and their functionality are expected to be operational. Thank you for your patience.

5. resolved Jun 13, 2025, 02:11 AM UTC

   This incident has been resolved.

6. postmortem Jun 13, 2025, 08:49 AM UTC

   Hello CVaaS Customers, We experienced a significant service disruption on June 12th, 2025, that lasted approximately 2 hours and 5 minutes. Our service regions were impacted due to a disruption of services delivered by our cloud service provider, and not due to a security-related incident. There was no data loss during the impact window. Despite this, parts of our service and features were impacted, most notably in our Network Provisioning services, including Studios features. This was due to the nature of the underlying cloud service provider disruption. Service regions cv-prod-us-central1-b and cv-prod-us-central1-c experienced the most pronounced impact due to an unfortunate set of circumstances where deployments of new changes coincided with our cloud service provider's disruptions. We continue to re-evaluate the designs and decisions made for CVaaS, and that, of course, includes our resiliency to disruptions like this. Our team will retrospect and act on systemic changes needed that arose from this event so we can continue to provide our users with a great product. Again, we deeply apologize for the disruption to CVaaS that you've experienced today, and thank you, as always, for your patience.