SearchStax incident

High number of Heartbeat alerts

Notice Resolved View vendor source →

SearchStax experienced a notice incident on April 1, 2021 affecting Pulse Monitoring, lasting 3h 5m. The incident has been resolved; the full update timeline is below.

Started
Apr 01, 2021, 10:13 PM UTC
Resolved
Apr 02, 2021, 01:19 AM UTC
Duration
3h 5m
Detected by Pingoru
Apr 01, 2021, 10:13 PM UTC

Affected components

Pulse Monitoring

Update timeline

  1. investigating Apr 01, 2021, 10:13 PM UTC

    Our team has observed a high number of heartbeat alerts with Azure Deployments. Our initial investigation shows that this is because of issues with Azure DNS: https://status.azure.com/en-gb/status.

  2. monitoring Apr 01, 2021, 10:58 PM UTC

    All existing heartbeat alerts have cleared out. Azure status page still reports Network Infrastructure Issues. Our team is closely monitoring the issue, but we do not see any ongoing issues with SearchStax deployments.

  3. resolved Apr 02, 2021, 01:19 AM UTC

    This incident has been marked as resolved. Impact of the Incident: * SearchStax Solr deployments were not impacted by this incident. * Since Pulse agents contact app.searchstax.com from the Azure nodes, they were impacted by the Azure DNS outage. As Pulse service did not receive any data from them over a time period, heartbeat alerts were reported.