SearchStax experienced a major incident on May 11, 2023 affecting Pulse Monitoring, lasting 4h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 11, 2023, 02:40 PM UTC
We have discovered false Heartbeat alerts for several deployments starting May 11, 2023,2:20pm UTC. Our team is investigating the issue. Our team has separate monitoring that is set up for all deployments for which they get alerted on if a node or deployment is unavailable. At this time, we checked all alerts triggered via Pulse and have confirmed that the deployments for which these alerts were triggered are all in a healthy state.
- resolved May 11, 2023, 06:53 PM UTC
Due to the recent fixes that were done for the April 24th incident with Pulse, our services had an issue updating some of the servers today that caused False Heartbeat alerts to go out. Our team has resolved the issue and put measures in place to prevent this in the future.