Multiple Azure regions facing Networking Issues

Update timeline

1. identified Jan 25, 2023, 08:47 AM UTC

   Multiple regions in Azure are facing Networking Issue, due to which the Solr deployments are unreachable - https://status.azure.com/en-us/status If your deployment is affected due to this issue, but has a DR configured, you should be moved over to the DR deployment. Our team is closely monitoring azure and will keep you updated on the status.

2. identified Jan 25, 2023, 09:10 AM UTC

   Deployments in most regions are now showing as up. However, we are still seeing an issue with the Azure West Europe region. If you have a DR and the DR was activated, our team has already contacted you vis support. We will be waiting for the Azure incident to resolve before we switch you back from DR. Update from Azure: "WarningAzure Networking - Multiple regions - Investigating Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in multiple regions, as well as other Microsoft services. We are actively investigating and will share updates as soon as more is known. This message was last updated at 08:53 UTC on 25 January 2023"

3. identified Jan 25, 2023, 09:21 AM UTC

   Our monitoring is now showing that all deployments are now reachable. Azure status page has not been updated. Our team will continue to monitor their status page and contact you to start switching the DRs back once Azure updates on their site.

4. identified Jan 25, 2023, 09:34 AM UTC

   Azure has updated the following: "We've determined the network connectivity issue is occurring with devices across the Microsoft Wide Area Network (WAN). This impacts connectivity between clients on the internet to Azure, as well as connectivity between services in datacenters, as well as ExpressRoute connections. The issue is causing impact in waves, peaking approximately every 30 minutes."

5. identified Jan 25, 2023, 09:41 AM UTC

   Update from Azure: "We have identified a recent WAN update as the likely underlying cause, and have taken steps to roll back this update. Our latest telemetry shows signs of recovery across multiple regions and services, and we are continuing to actively monitor the situation." As per this update, our team with contact you to start switching the DRs back to the primary deployment if the DR was activated.

6. monitoring Jan 25, 2023, 10:06 AM UTC

   All DRs have been switched back to Primary. Azure reports: "Our telemetry shows consistent signs of recovery from 09:00 UTC onwards across multiple regions and services, and we are continuing to actively monitor the situation. With WAN networking now seeing recovery, we are working to ensure full recovery for impacted services."

7. resolved Jan 25, 2023, 02:01 PM UTC

   This incident has been resolved.