Is HetrixTools down?

We've been internally alerted of a higher-than-usual fail rate for our Uptime Monitoring system. The DENIC domain registry is currently having issues, causing most of the .DE domains to not resolve. Our system is currently detecting these uptime monitors as being down, as it should be, since they are not resolving properly. Sources: https://status.denic.de/ https://news.ycombinator.com/item?id=48027897 https://www.reddit.com/r/de_EDV/comments/1t4qlrg/psa_die_dezone_l%C3%B6st_gerade_gro%C3%9Ffl%C3%A4chig_nicht_mehr/ This issue is out of our hands, and our system is working as expected. We're posting this here so that our users are informed of what is happening with their .DE uptime monitors. We'll continue to monitor the situation as it progresses.

This incident has been resolved.

At around 13:12 UTC, we noticed an increase in errors with our Server Monitoring Metrics API. After further investigation, our techs discovered that one of our back-end nodes was experiencing issues and not ingesting all monitoring metrics as expected. The issue has been fully resolved by 14:00 UTC. The issue has affected about 2.7% of our Server Monitoring Agents, which may have had issues posting their metrics to our platform.

We're currently investigating issues that are affecting our Blacklist Monitoring Queue.

The issue has been identified and a fix is being implemented.

A fix has now been implemented, and we're slowly restarting the microservices powering the Blacklist Monitoring Queue.

The queue has a big backlog to process, so it will take some time for it to catch up and process monitors in real time, as it does when healthy. We'll provide further updates here as this process advances.

The queue backlog is now at 50% of its level when processing was resumed. We'll follow up with another update once it's fully caught up.

The backlog has now been fully processed, and the Blacklist Monitoring Queue is now running healthy as expected.

We've been observing an increased rate of 520 HTTP errors from HEAD requests to websites behind Cloudflare. Cloudflare has recently acknowledged the issue on their end and is working on a fix: https://www.cloudflarestatus.com/incidents/kxvggpg7kwx5 In the meantime, you can fix the issue right away in your HetrixTools account by changing the "HTP Method" from "HEAD" to "GET" for any of your website monitors (by editing the monitor's settings): https://docs.hetrixtools.com/wp-content/uploads/2022/10/image.png You can check if you're affected by this Cloudflare issue by checking your Location Fail Log (https://docs.hetrixtools.com/location-fail-log/) and looking for 520 HTTP errors.

Cloudflare has updated the status of their incident to resolved. We're noticing that the number of 520 HTTP errors from HEAD requests has gone down. This incident is now resolved.