HetrixTools Outage History

The issue has been identified: Microsoft SNDS silently changed how its API responds to data requests, breaking most of our SNDS data sync integration. We're currently working on implementing a fix for this issue and forcing a new data sync after.

We've been internally alerted of a higher-than-usual fail rate for our Uptime Monitoring system. The DENIC domain registry is currently having issues, causing most of the .DE domains to not resolve. Our system is currently detecting these uptime monitors as being down, as it should be, since they are not resolving properly. Sources: https://status.denic.de/ https://news.ycombinator.com/item?id=48027897 https://www.reddit.com/r/de_EDV/comments/1t4qlrg/psa_die_dezone_l%C3%B6st_gerade_gro%C3%9Ffl%C3%A4chig_nicht_mehr/ This issue is out of our hands, and our system is working as expected. We're posting this here so that our users are informed of what is happening with their .DE uptime monitors. We'll continue to monitor the situation as it progresses.

The issue has been identified and a fix is being implemented.

We've been observing an increased rate of 520 HTTP errors from HEAD requests to websites behind Cloudflare. Cloudflare has recently acknowledged the issue on their end and is working on a fix: https://www.cloudflarestatus.com/incidents/kxvggpg7kwx5 In the meantime, you can fix the issue right away in your HetrixTools account by changing the "HTP Method" from "HEAD" to "GET" for any of your website monitors (by editing the monitor's settings): https://docs.hetrixtools.com/wp-content/uploads/2022/10/image.png You can check if you're affected by this Cloudflare issue by checking your Location Fail Log (https://docs.hetrixtools.com/location-fail-log/) and looking for 520 HTTP errors.

We're aware that some of our components behind Cloudflare are experiencing issues at this time: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 This currently affects our services as follows: Our Blacklist Monitoring service remains unaffected. Our Uptime Monitoring service continues to process a high number of detected outages, so notifications may be delayed as it processes through its backlog. The Server Agent Metrics system is impaired, as it is behind Cloudflare.

Unfortunately, the outage is still ongoing at this time. We'll continue to monitor the situation as it progresses. Further info about this outage can be found on Cloudflare's status page: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

We’ve identified an issue in our Linux Server Monitoring Agent that occurs when Daylight Saving Time ends (in autumn). If the server where the agent is installed is not configured to use the UTC timezone, the agent will fail to collect metrics during the repeated (backward) hour. This only affects the following versions of our Linux Server Monitoring Agent: 2.3.1 2.3.2 2.3.3 2.3.4 The issue stems from how the agent scheduler uses the server's time to run the agent every one minute; a scheduler that won't fire up when the hour turns backwards. A solution has been implemented for this with version 2.3.5 of our Linux Server Monitoring Agent. You can easily update your agent with the following command: wget -4 -qO- https://raw.githubusercontent.com/hetrixtools/agent/master/hetrixtools_update.sh | sudo bash

We have identified the issue and are currently working on a fix for it. Impact: Daily/Weekly/Monthly Reports and Blacklist Monitoring Notifications may be delayed while we fix the issue.

We implemented a fix, and the Email Queue is now functioning properly and catching up on its backlog. We're monitoring it to make sure everything works as expected.

The incident has been identified and confirmed by AWS, as per their status page: https://health.aws.amazon.com/health/status We're working on mitigating the impact on our end while they work on solving the root cause of the problem.

Cloudflare's DNS servers (1.1.1.1 and 1.0.0.1) seem to be having major issues: https://www.cloudflarestatus.com/incidents/28r0vbbxsh8f https://x.com/Newspicel/status/1944880481704796373 This issue affects our Server Monitoring Agent installed on servers that use these DNS servers. Since these servers cannot resolve any hostnames at this time, our Server Monitoring Agent cannot send any metrics to our platform. Resolution: Use diverse DNS servers on your servers (i.e., a mixture of Cloudflare and Google DNS servers), or wait for Cloudflare to fix its DNS servers so that your server regains full DNS resolution capabilities.