SearchStax experienced a minor incident on March 29, 2020 affecting Admin App, lasting 9d 14h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 29, 2020, 07:13 AM UTC
Our team noticed a large number of errors in Heartbeat detection for deployments for the alerts setup via SearchStax Pulse. At this time, we checked all alerts triggered via Pulse and have confirmed that the deployments for which these alerts were triggered are all in a healthy state. We have temporarily paused the alert emails while we investigate the issue. Our team has separate monitoring that is set up for all deployments for which they get alerted on if a node or deployment is unavailable. If you are one of our Premium Customers, and if we encounter any issue, our team will reach out to you via email. As there is more progress on the issue, we will keep updating this incident.
- identified Apr 01, 2020, 12:41 AM UTC
Our team has performed Maintenance and Upgrades on our Pulse Service. We have internally turned on heartbeat alerts, but the email notifications and webhooks are disabled as we monitor the service. Because of this incident, there are SearchStax monitoring agents on some deployments that are not working and need to be restarted. If your deployment's SearchStax monitoring agent is impacted, our team will be contacting you for performing maintenance activities on these deployments in the next few days.
- monitoring Apr 05, 2020, 07:48 PM UTC
Our team has completed maintenances on deployments where SearchStax Monitoring Agent was unresponsive. We have now enabled email and webhook notifications. The Pulse Service now should be fully functional on all deployments. We are continuing to monitor the status to identify if any false alerts do occur in the next 24-72 hours. If there are no false alerts in that period, we will mark this issue as resolved. Thanks for your patience as we work through this incident.
- resolved Apr 07, 2020, 09:48 PM UTC
We have been monitoring heartbeat alerts closely for the last 48 hours and no false alerts were generated during this period. We will proceed to mark this incident as resolved.