SearchStax incident

Multiple Solr Deployments in Azure - Central US Region Degraded or Down

Major Resolved View vendor source →

SearchStax experienced a major incident on July 18, 2024 affecting Customer Dedicated Deployments, lasting 17h 16m. The incident has been resolved; the full update timeline is below.

Started
Jul 18, 2024, 10:44 PM UTC
Resolved
Jul 19, 2024, 04:01 PM UTC
Duration
17h 16m
Detected by Pingoru
Jul 18, 2024, 10:44 PM UTC

Affected components

Customer Dedicated Deployments

Update timeline

  1. identified Jul 18, 2024, 10:44 PM UTC

    Azure has an ongoing issue in Central US region, affecting multiple Solr deployments in that region. If you are one of our Hot or Warm DR Plans, and your deployment is affected, your DR should now be activated. Our team is actively monitoring the status and Cold DRs will be activated based on RTOs. Message from Azure: -------------------- Impact Statement: Starting at 21:56 UTC on 18 Jul 2024, you have been identified as a customer using Virtual Machines in Central US who may experience connection failures when trying to access some Virtual Machines hosted in the region. These Virtual Machines may have also restarted unexpectedly. Current Status: We are aware of this issue and are actively investigating. An update will be provided as events warrant.

  2. identified Jul 19, 2024, 12:03 AM UTC

    Azure team is still investigating the issue. The status is now available on Azure Status page here: https://azure.status.microsoft/en-us/status/

  3. monitoring Jul 19, 2024, 03:32 AM UTC

    Azure has reported on https://azure.status.microsoft/en-us/status/ that it has started applying the mitigation steps and we are seeing signs of recovery. All SearchStax Premium deployments in the Central US Region have recovered. If a Disaster Recovery plan was activated for any of these deployments, the traffic has now been moved back to the Primary deployment. If we continue monitoring this issue till it gets closed from Azure end.

  4. resolved Jul 19, 2024, 04:01 PM UTC

    This incident has been resolved.