Fire had 22 outages in the last 2 years totaling 32m of downtime — averaging 0.9 incidents per month.
There were 22 Fire outages since November 13, 2024 totaling 32m of downtime. Each is summarised below — incident details, duration, and resolution information.
Hetzner has reported a fault on the node that Fire is hosted on. This outage is out of Fire's control but once resolved, everything should come back automatically
Health checks have recovered and Fire is back online along with Hetzner stating the incident has been resolved
Type: Maintenance Duration: 3 minutes Affected Components: Aether, Website, Bot May 18, 16:53:00 GMT+0 - Identified - Fire & some related services will be unavailable for a short period of time while some updates are installed May 18, 16:53:01 GMT+0 - Identified - Maintenance is now in progress May 18, 16:55:45 GMT+0 - Completed - Maintenance has completed successfully.
Type: Incident Duration: 35 minutes Affected Components: Beta Bot Apr 9, 01:05:00 GMT+0 - Identified - Fire Beta is currently unavailable due to an incident at Hetzner resulting in the server being offline Apr 9, 01:40:00 GMT+0 - Resolved - This incident has been resolved and services have returned to normal operation.
Type: Incident Duration: 15 minutes Affected Components: Beta Bot Mar 15, 16:41:11 GMT+0 - Investigating - Beta Bot cannot be accessed at the moment. This incident was created by an automated monitoring service. Mar 15, 16:56:12 GMT+0 - Resolved - Beta Bot is now operational! This update was created by an automated monitoring service.
Type: Incident Duration: 41 minutes Affected Components: Beta Bot, Bot Mar 12, 20:30:00 GMT+0 - Identified - Fire is currently experiencing elevated latency & error rates due to issues with Discord's API. Discord has posted about the issue on their own status page, [https://discordstatus.com/incidents/](https://discordstatus.com/incidents/dsv750sj44fn) Mar 12, 20:59:52 GMT+0 - Monitoring - Discord has implemented a fix and both error rates & latency have returned closer to expected levels. I will continue monitoring for some time to ensure things are fully resolved Mar 12, 21:10:56 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 29 minutes Affected Components: Aether, , Website, Beta Bot, Bot, Backend → Feb 25, 21:08:32 GMT+0 - Investigating - It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console Feb 25, 21:13:12 GMT+0 - Monitoring - Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning Feb 25, 21:37:39 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 1 hour and 40 minutes Affected Components: Bot Feb 3, 15:35:46 GMT+0 - Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable Feb 3, 17:15:46 GMT+0 - Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues
Type: Incident Duration: 1 hour and 1 minute Affected Components: Aether, Bot, Website Dec 31, 08:24:21 GMT+0 - Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable Dec 31, 08:29:07 GMT+0 - Investigating - As part of some internal restructuring, it seems that the websocket connection to Aether is not working correctly, preventing the bot from launching correctly. I'm currently investigating this and hope to get this resolved as soon as possible Dec 31, 08:36:52 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Dec 31, 09:25:19 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 39 minutes Affected Components: Bot Dec 19, 23:00:38 GMT+0 - Investigating - Many of Fire's API requests to Discord are currently failing indicating a potential outage. I have had to shut down one of Fire's clusters as it got stuck in a loop retrying requests due to this issue and am working on a solution so that it can be restored Dec 19, 23:22:38 GMT+0 - Monitoring - Things look to have improved, requests aren't failing nearly as often and latency has returned to normal levels. I will continue to monitor requests for some time just to be sure it's fully resolved Dec 19, 23:39:55 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 7 hours Affected Components: Aether, Hastebin, Bot, Website Nov 18, 18:52:18 GMT+0 - Resolved - Cloudflare have stated that things are now operating normally Nov 18, 11:52:30 GMT+0 - Identified - Cloudflare's network is currently experiencing issues which will affect Fire due to both it and Discord making use of Cloudflare's services You may experience slow responses or failed interactions when attempting to use commands in the bot and web services may return 5XX errors
Type: Maintenance Duration: 44 minutes Affected Components: Bot, Website Nov 10, 15:00:00 GMT+0 - Identified - During this time, the new website will be deployed alongside some updates to the bot. This is not expected to take long nor should there be any problems as both the website & bot changes have been tested locally & in staging The website will be unavailable for a brief period of time while a production build is deployed and configured while the bot should only be unavailable for a few seconds while it restarts for a normal deploy Nov 10, 15:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 10, 15:07:14 GMT+0 - Identified - The bot has been updated so now the website is going to be changed over Nov 10, 15:23:21 GMT+0 - Identified - The new website has been successfully deployed and final checks are being ran to ensure everything is working as intended Nov 10, 15:43:42 GMT+0 - Completed - Maintenance has completed successfully.
Type: Incident Duration: 16 minutes Affected Components: Bot Jun 22, 12:04:55 GMT+0 - Investigating - Quotes are currently disabled while an issue is being investigated Jun 22, 12:17:18 GMT+0 - Identified - The issue has been identified and a fix is being made Jun 22, 12:21:22 GMT+0 - Resolved - The fix has been deployed and quotes have been re-enabled
Type: Incident Duration: 14 minutes Affected Components: Website, Aether, Bot May 14, 00:49:50 GMT+0 - Identified - Discord turned 10 years old and is now old and slow. Rest latency started to increase at 00:20 UTC but not enough to trigger alerts so I only discovered things are broken a few minutes ago and seems like Discord did too as they only posted about it now too May 14, 00:58:19 GMT+0 - Monitoring - Discord claims that things are fixed but I don't trust them so this will stay open a little longer while I keep an eye on boring graphs to see if it goes stonks (but in a bad way) again May 14, 01:04:04 GMT+0 - Resolved - Some requests (primarily adding roles to members) are still showing somewhat higher latency than normal but things are generally fine and Fire should be functioning normally
Type: Incident Affected Components: Aether Feb 23, 14:39:00 GMT+0 - Resolved - An issue has just been resolved where the [inv.wtf](http://inv.wtf) domain was returning 404s for all requests due to a mistake in an earlier deploy. Existing monitoring was unfortunately unable to detect the issue and fire alerts which meant it went unnoticed for longer than it should have. Apologies for any inconvenience caused. I will be looking into improving monitoring to detect these kinds of issues and ensure they do not happen (or at least not for as long) in the future
Type: Incident Duration: 48 minutes Affected Components: Bot, Website, Aether, Hastebin Feb 1, 22:43:31 GMT+0 - Identified - It looks like Fire's VPS has ran out of available space. I knew this was approaching but I thought I had more time than I did. I have shut it down for now and will attempt to increase the partition size (as the disk itself had already been increased) and then bring it back online Feb 1, 23:07:33 GMT+0 - Monitoring - Fire now has about double the storage it used to and after a slight hiccup with InfluxDB not wanting to start, everything should be back online. I will be monitoring the situation alongside trying to figure out what caused the sudden increase in disk space (my current suspicion is InfluxDB considering the trouble I had getting it to come back) Feb 1, 23:31:38 GMT+0 - Resolved - This incident has been resolved.