Resolved after 19 minutes, on March 28, 2020.
# Postmortem: Downtime 🔨 Hi all, Campfire’s Development Operations Manager, Sam here with a postmortem for the issues experienced here. ## What went wrong? We’ve been looking at our history of downtime like this and we’ve come to the conclusion that we’re simply putting a little too much load on our London server. This likely is causing the server to “burst” \(run out of CPU credits\) and therefore is powering down our instance. We’d have no issue believing this, but the issue is it’s not actually powering down. It’s becoming entirely unresponsive. We’ve taken screenshots via the AWS EC2 manager and the instance looks perfectly normal. ## What do we 100% know? We know that the server became overloaded \(at about 99%\) CPU utilisation and then went completely silent. This therefore leads us to believe it’s a performance issue. ## What are we doing about this? We’re doing a few things. 1. We’re moving our EC2 instance to America, so it has a quicker response time with the Discord gateway \(this therefore means less CPU time and better performance\) 2. We’re investing more money into compute power to increase our cloud footprint by at least 200%. 3. We’re trying out a different process manager. We previously were using forever. We’re now trialing PM2 based on a recommendation from a third party who develops bots and services in our space. We don’t anticipate to have any more issues like this, but naturally we may have some again in the future. We are trying our best and training staff on procedures for our downtime so we can handle it in the best way possible. Many thanks for your continued support.
Update timeline
- investigating Mar 28, 2020, 04:04 PM UTC
We are currently investigating this issue.
- investigating Mar 28, 2020, 04:07 PM UTC
This issue appears to have been present since UTC 13:55. We're now launching an investigation with our Engineering Ops Team. We'll keep you updated as this case develops. Our main priority is to restore service.
- identified Mar 28, 2020, 04:11 PM UTC
The issue has been identified as a large amount of stress on our CPU causing it to terminate our instance. We're now looking into what caused this large CPU spike. CPU Graph: https://cdn.discordapp.com/attachments/552944213768011816/693491730477088858/unknown.png
- identified Mar 28, 2020, 04:12 PM UTC
We're now beginning to restore all bots running on this instance.
- monitoring Mar 28, 2020, 04:18 PM UTC
All bots have been restored and we are now monitoring the issue to ensure that things run smoothly.
- monitoring Mar 28, 2020, 04:18 PM UTC
We are continuing to monitor for any further issues.
- resolved Mar 28, 2020, 04:24 PM UTC
All looks healthy! We're closing this incident now. So sorry about the issues this caused. A post mortem may be published soon.
- postmortem Mar 29, 2020, 12:38 AM UTC
# Postmortem: Downtime 🔨 Hi all, Campfire’s Development Operations Manager, Sam here with a postmortem for the issues experienced here. ## What went wrong? We’ve been looking at our history of downtime like this and we’ve come to the conclusion that we’re simply putting a little too much load on our London server. This likely is causing the server to “burst” \(run out of CPU credits\) and therefore is powering down our instance. We’d have no issue believing this, but the issue is it’s not actually powering down. It’s becoming entirely unresponsive. We’ve taken screenshots via the AWS EC2 manager and the instance looks perfectly normal. ## What do we 100% know? We know that the server became overloaded \(at about 99%\) CPU utilisation and then went completely silent. This therefore leads us to believe it’s a performance issue. ## What are we doing about this? We’re doing a few things. 1. We’re moving our EC2 instance to America, so it has a quicker response time with the Discord gateway \(this therefore means less CPU time and better performance\) 2. We’re investing more money into compute power to increase our cloud footprint by at least 200%. 3. We’re trying out a different process manager. We previously were using forever. We’re now trialing PM2 based on a recommendation from a third party who develops bots and services in our space. We don’t anticipate to have any more issues like this, but naturally we may have some again in the future. We are trying our best and training staff on procedures for our downtime so we can handle it in the best way possible. Many thanks for your continued support.
Your product runs on more than Campfire HQ
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services