Fire Outage History

Type: Incident Duration: 35 minutes Affected Components: Beta Bot Apr 9, 01:05:00 GMT+0 - Identified - Fire Beta is currently unavailable due to an incident at Hetzner resulting in the server being offline Apr 9, 01:40:00 GMT+0 - Resolved - This incident has been resolved and services have returned to normal operation.

Type: Incident Duration: 15 minutes Affected Components: Beta Bot Mar 15, 16:41:11 GMT+0 - Investigating - Beta Bot cannot be accessed at the moment. This incident was created by an automated monitoring service. Mar 15, 16:56:12 GMT+0 - Resolved - Beta Bot is now operational! This update was created by an automated monitoring service.

Type: Incident Duration: 41 minutes Affected Components: Beta Bot, Bot Mar 12, 20:30:00 GMT+0 - Identified - Fire is currently experiencing elevated latency & error rates due to issues with Discord's API. Discord has posted about the issue on their own status page, [https://discordstatus.com/incidents/](https://discordstatus.com/incidents/dsv750sj44fn) Mar 12, 20:59:52 GMT+0 - Monitoring - Discord has implemented a fix and both error rates & latency have returned closer to expected levels. I will continue monitoring for some time to ensure things are fully resolved Mar 12, 21:10:56 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 29 minutes Affected Components: Aether, , Website, Beta Bot, Bot, Backend → Feb 25, 21:08:32 GMT+0 - Investigating - It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console Feb 25, 21:13:12 GMT+0 - Monitoring - Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning Feb 25, 21:37:39 GMT+0 - Resolved - This incident has been resolved.

Type: Maintenance Duration: 2 hours and 26 minutes Affected Components: Aether, , Website, Bot, Backend → Feb 18, 11:00:01 GMT+0 - Identified - Maintenance is now in progress Feb 18, 11:28:56 GMT+0 - Identified - Fire is now back online and now running in Kubernetes! Next up will be reconfiguring DNS records which may take some time due to SSL certificate provisioning. This means that web services will still be unavailable for a while Feb 18, 11:00:00 GMT+0 - Identified - During this time, we will be performing upgrades to Fire's infrastructure, changing how the bot and related applications are deployed/managed on the machine. These upgrades have already been tested in a production-like environment by moving Fire Beta to a separate machine for testing & learning. The main upgrade should realistically only take 15-20 minutes with most of the time being spent replacing updating configurations for services exposed to the web as they can no longer be simply pointed to `localhost` & some port and even then, I don't expect this to take the entire duration of the maintenance but I'd rather schedule a larger window just in case For anyone interested, I am moving Fire away from running directly on the machine and being managed via PM2 to using Docker containers & Kubernetes. This move will make it much easier to keep Fire up to date as upgrading the OS or Node version is as simple as changing the image used. It also comes with a more maintainable solution for offsite backups as I've configured the Postgres cluster with automatic backups to CloudFlare R2 rather than moving them to my Mac every so often Feb 18, 12:05:31 GMT+0 - Identified - Everything should be up and running now with all DNS records recreated with their new values. Some users may experience issues due to delays in DNS propagation and SSL certificate provisioning but these should eventually resolve themselves Extensive review & monitoring will be taking place for the next hour or so to ensure nothing has been missed and to do some cleanup of older resources Feb 18, 13:25:40 GMT+0 - Completed - Maintenance has completed successfully. There are a few issues with certain features/tooling but they're minor enough that I'm willing to let them be for now and will work on them within the next few days

Type: Incident Duration: 1 hour and 40 minutes Affected Components: Bot Feb 3, 15:35:46 GMT+0 - Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable Feb 3, 17:15:46 GMT+0 - Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues