Fire had 21 outages in the last 2 years totaling 16h 13m of downtime — averaging 0.9 incidents per month.
There were 21 Fire outages since November 13, 2024 totaling 16h 13m of downtime. Each is summarised below — incident details, duration, and resolution information.
Hetzner has reported a fault on the node that Fire is hosted on. This outage is out of Fire's control but once resolved, everything should come back automatically.
Health checks have recovered and Fire is back online along with Hetzner stating the incident has been resolved.
Fire Beta is currently unavailable due to an incident at Hetzner resulting in the server being offline .
This incident has been resolved and services have returned to normal operation..
Beta Bot cannot be accessed at the moment. This incident was created by an automated monitoring service..
Beta Bot is now operational! This update was created by an automated monitoring service..
Fire is currently experiencing elevated latency & error rates due to issues with Discord's API. Discord has posted about the issue on their own status page, [https://discordstatus.com/incidents/](https://discordstatus.com/incidents/dsv750sj44fn).
Discord has implemented a fix and both error rates & latency have returned closer to expected levels. I will continue monitoring for some time to ensure things are fully resolved.
This incident has been resolved..
It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console.
Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning.
This incident has been resolved..
Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.
Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.
Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.
As part of some internal restructuring, it seems that the websocket connection to Aether is not working correctly, preventing the bot from launching correctly. I'm currently investigating this and hope to get this resolved as soon as possible.
We implemented a fix and are currently monitoring the result..
This incident has been resolved..
Many of Fire's API requests to Discord are currently failing indicating a potential outage. I have had to shut down one of Fire's clusters as it got stuck in a loop retrying requests due to this issue and am working on a solution so that it can be restored.
Things look to have improved, requests aren't failing nearly as often and latency has returned to normal levels. I will continue to monitor requests for some time just to be sure it's fully resolved.
This incident has been resolved..
Cloudflare's network is currently experiencing issues which will affect Fire due to both it and Discord making use of Cloudflare's services You may experience slow responses or failed interactions when attempting to use commands in the bot and web services may return 5XX errors.
Cloudflare have stated that things are now operating normally.
Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.
Upon investigating, it appears that the issues Discord are having at the moment have resulted in the gateway connections for Fire's clusters getting stuck in a form of limbo state where it appears to be connected but nothing is being received (and I have no way to test if anything can be sent as I'd normally do that via a command) I will attempt to restart clusters, starting with the cluster that has the least unavailable servers (cluster 3 with only a single unavailable server), since servers being unavailable on connect can slow down startup and might result in further issues.
The restart was successful meaning cluster 3 is now online and functioning. I will continue going in order of least unavailable servers as that seems like the best thing to do right now.
Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.
All clusters are back online, however, they may have slow responses or other issues related to the Discord outage. Monitoring will continue until Discord has marked their incident as resolved.
This incident has been resolved..
Quotes are currently disabled while an issue is being investigated.
The issue has been identified and a fix is being made.
The fix has been deployed and quotes have been re-enabled.
Discord turned 10 years old and is now old and slow. Rest latency started to increase at 00:20 UTC but not enough to trigger alerts so I only discovered things are broken a few minutes ago and seems like Discord did too as they only posted about it now too.
Discord claims that things are fixed but I don't trust them so this will stay open a little longer while I keep an eye on boring graphs to see if it goes stonks (but in a bad way) again.
Some requests (primarily adding roles to members) are still showing somewhat higher latency than normal but things are generally fine and Fire should be functioning normally.
Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.
It seems that Discord may be having issues with the gateway at the moment. All of Fire's websocket connections were dropped simultaneously and are unable to reconnect along with my own websocket connection in the app experiencing the same issue There are also many reports flooding in on Twitter about issues connecting so it doesn't seem like I or Fire are alone in this..
Shard 3 has been able to successfully reconnect so the rest should hopefully follow suit but unfortunately I am still unable to get in myself and therefore unable to verify if that shard is functioning correctly.
Shard 2 has now been able to reconnect. I tried restarting cluster 0 (shard 0) to see if that would help but it is still stuck in the same loop as shard 1.
Shard 0 has risen from the dead, now just waiting on shard 1....
Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.
An issue has just been resolved where the [inv.wtf](http://inv.wtf) domain was returning 404s for all requests due to a mistake in an earlier deploy. Existing monitoring was unfortunately unable to detect the issue and fire alerts which meant it went unnoticed for longer than it should have. Apologies for any inconvenience caused. I will be looking into improving monitoring to detect these kinds of issues and ensure they do not happen (or at least not for as long) in the future.
It looks like Fire's VPS has ran out of available space. I knew this was approaching but I thought I had more time than I did. I have shut it down for now and will attempt to increase the partition size (as the disk itself had already been increased) and then bring it back online.
Fire now has about double the storage it used to and after a slight hiccup with InfluxDB not wanting to start, everything should be back online. I will be monitoring the situation alongside trying to figure out what caused the sudden increase in disk space (my current suspicion is InfluxDB considering the trouble I had getting it to come back).
This incident has been resolved..