Socio Outage History

Started Jun 12, 2025, 06:20 PM UTC· resolved Jun 13, 2025, 04:09 AM UTC

1. investigating Jun 12, 2025, 06:20 PM UTC

   We’re currently experiencing a service disruption with our Platform, Web App, and Mobile App due to Firebase related incident.

2. identified Jun 12, 2025, 06:30 PM UTC

   Recent incidents (2) Last updated time: 12 Jun 2025, 11:27 PDT Refresh Google Cloud Outage is blocking all tests in Test Lab Active Start Time: 12 Jun 2025, 11:10 PDT Last update: 12 Jun 2025, 11:25 PDT Impacted products: Test Lab Firebase App Hosting API and Origin Outage Active Start Time: 12 Jun 2025, 11:15 PDT Last update: 12 Jun 2025, 11:24 PDT Impacted products: App Hosting

3. monitoring Jun 12, 2025, 07:07 PM UTC

   Firebase side seems operational, we will be monitoring for the performance of the systems.

4. monitoring Jun 12, 2025, 07:07 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Jun 13, 2025, 04:09 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 24, 2024, 07:00 PM UTC· resolved Oct 24, 2024, 08:42 PM UTC

1. resolved Oct 25, 2024, 06:59 PM UTC

   Webex Events infrastructure depends on AWS and they had increased delays associating Route 53 Private Hosted Zones to VPC. Our customers had intermittent issues accessing the Platform and the Web App. This is their explanation on the status page: ----------- Oct 24 12:01 PM PDT Beginning around 10:05 AM PDT customers began to see issues with creating new hosted zone to VPC associations for Route 53 Private Hosted Zones. This issue does not affect Public or Private DNS record updates, which are working normally. All DNS queries are also being answered at this time. This issue is also affecting certain workloads for some other AWS services: Elastic Kubernetes Service ClusterCreate operations are resulting in errors. There is no impact to existing EKS clusters. Parallel Computing Service (PCS) CreateCluster operations are resulting in errors. There is no impact to existing PCS clusters. VPC PrivateLink is experiencing delays for Private DNS enabled interface VPC Endpoints for CreateVpcEndpoint and ModifyVpcEndpoint API calls in all regions. Existing PrivateLink resources are working as normal. HealthOmics is experiencing increased error rates for ongoing runs for both STATIC and DYNAMIC run storage types. Engineers are engaged on the issue and are actively working towards mitigation. We will provide another update within 30 minutes. ----------- Oct 24 12:36 PM PDT We have identified the root cause of the issue causing delays creating new hosted Zone to VPC associations. Engineers are currently engaged and working on multiple mitigation paths. We have confirmed that this issue only affects new hosted zone to VPC associations for Route 53 Private Hosted Zones. This issue does not affect Public or Private DNS record updates, which are working normally, and all DNS queries are being answered at this time. Elastic Kubernetes Service CreateCluster operations are resulting in errors. There is no impact to existing EKS clusters. Parallel Computing Service (PCS) CreateCluster operations are resulting in errors. There is no impact to existing PCS clusters. VPC PrivateLink is experiencing delays for Private DNS enabled interface VPC Endpoints for CreateVpcEndpoint and ModifyVpcEndpoint API calls in all regions. Existing PrivateLink resources are working as normal. HealthOmics is experiencing increased error rates for ongoing runs for both STATIC and DYNAMIC run storage types. FSx is experiencing issues affecting the ability to connect to recently-created Amazon FSx file systems. Existing file systems, and customers connecting to file systems using IP addresses directly, are not impacted. We will provide another update within 30 minutes. ----------- Oct 24 1:05 PM PDT Work is ongoing to mitigate the issue causing delays creating new hosted Zone to VPC associations. Engineers continue to pursue multiple mitigation paths. We want to reassure customers that this issue does not affect Public or Private DNS record updates, which are working normally, and all DNS queries are being answered at this time. Elastic Kubernetes Service CreateCluster operations are resulting in errors. There is no impact to existing EKS clusters. Parallel Computing Service (PCS) CreateCluster operations are resulting in errors. There is no impact to existing PCS clusters. VPC PrivateLink is experiencing delays for Private DNS enabled interface VPC Endpoints for CreateVpcEndpoint and ModifyVpcEndpoint API calls in all regions. Existing PrivateLink resources are working as normal. HealthOmics is experiencing increased error rates for ongoing runs for both STATIC and DYNAMIC run storage types. FSx is experiencing issues affecting the ability to connect to recently-created Amazon FSx file systems. Existing file systems, and customers connecting to file systems using IP addresses directly, are not impacted. Based on the current status of the event we expect to provide an update within 60 minutes. ----------- Oct 24 1:45 PM PDT Between 10:05 AM and 1:42 PM PDT we experienced increased delays creating new hosted Zone to VPC associations. Engineers were engaged at 10:09 AM and began investigating. By 10:45 AM root cause was understood and the team set out on parallel mitigation courses. By 1:05 PM we began seeing the first signs of recovery and by 1:42 PM our telemetry indicated we were recovered. We want to reassure customers that this issue did not affect Public or Private DNS record updates, and all DNS queries were being answered throughout the course of the event. During this time, services which create and associate Hosted Zones to VPCs on behalf of customers also experienced impact. Examples of these services have been listed in previous updates to this Health Event. The issue is resolved and all services are operating normally.

Read the full incident report →