SearchStax incident

False Pulse Heartbeat Alerts

Notice Resolved View vendor source →

SearchStax experienced a notice incident on December 3, 2020 affecting Pulse Monitoring, lasting 1d 1h. The incident has been resolved; the full update timeline is below.

Started
Dec 03, 2020, 08:03 PM UTC
Resolved
Dec 04, 2020, 09:14 PM UTC
Duration
1d 1h
Detected by Pingoru
Dec 03, 2020, 08:03 PM UTC

Affected components

Pulse Monitoring

Update timeline

  1. investigating Dec 03, 2020, 08:03 PM UTC

    We have discovered false Heartbeat alerts for several deployments. Our team is investigating the issue. At this time, we checked all alerts triggered via Pulse and have confirmed that the deployments for which these alerts were triggered are all in a healthy state. Our team has separate monitoring that is set up for all deployments for which they get alerted on if a node or deployment is unavailable. If you are one of our Premium Customers, and if we encounter any issue, our team will reach out to you via email.

  2. identified Dec 03, 2020, 11:35 PM UTC

    Our team has reattached Pulse agents on deployments which should resolve the Heartbeat alerts. If we are unable to reattach the agent, our team will be contacting you for a window to restart the service.

  3. resolved Dec 04, 2020, 09:14 PM UTC

    Our team has restarted Solr/Zookeeper service on deployments after coordinating with the clients that were impacted. The incident has been resolved.