Resolved after 8 hours 56 minutes, affecting Web Interface and Web Interface and 1 more, on December 7, 2021.
The issues on AWS affected our product as well as Heroku and even Intercom, our main communication channel with our customers. In Cron To Go, new issues arised after the AWS issues were fixed since Cron To Go was executing jobs but still received errors from Heroku APIs and retried execution. This caused the job queue to bloat up more than the workers' capacity and caused jobs to run out of schedule \(due to retries\). We purged the queue and job execution is back to normal. Note that if jobs were meant to run during the incidents, they may have no executed at all so you may want to run them manually.
Affected components
Update timeline
- investigating Dec 07, 2021, 05:10 PM UTC
We are investigating availability issues for a significant portion of our platform due to issue with our upstream provider (AWS) in the us-east-1 region.
- identified Dec 07, 2021, 10:26 PM UTC
Our engineers have detected that Heroku and our upstream provider (AWS) are experiencing elevated error rates in the US region. This may impact opening our add-on dashboards, communicating with the API, Webhooks, authentication and more. We keep to work closely with them for a full path to mitigation.
- identified Dec 07, 2021, 10:28 PM UTC
AWS have identified the root cause of this issue to be an impairment of several network devices. They have executed a mitigation which is showing significant recovery in the US-EAST-1 Region. We still do not have an ETA for full recovery at this time.
- identified Dec 07, 2021, 10:38 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Dec 07, 2021, 10:54 PM UTC
AWS have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. All services are now independently working through service-by-service recovery. We will keep monitoring the recovery rates and keep you posted.
- monitoring Dec 07, 2021, 11:12 PM UTC
The AWS team is still working on getting their services fully recovered. We are using API Gateway, DynamoDB and EventBridge which are still experiencing impact. We will keep monitoring for these services recovery rates and keep updating accordingly.
- monitoring Dec 07, 2021, 11:21 PM UTC
We are continuing to monitor for any further issues.
- monitoring Dec 07, 2021, 11:33 PM UTC
We are continuing to monitor for any further issues.
- resolved Dec 08, 2021, 05:06 AM UTC
This incident has been resolved.
- postmortem Dec 08, 2021, 08:32 AM UTC
The issues on AWS affected our product as well as Heroku and even Intercom, our main communication channel with our customers. In Cron To Go, new issues arised after the AWS issues were fixed since Cron To Go was executing jobs but still received errors from Heroku APIs and retried execution. This caused the job queue to bloat up more than the workers' capacity and caused jobs to run out of schedule \(due to retries\). We purged the queue and job execution is back to normal. Note that if jobs were meant to run during the incidents, they may have no executed at all so you may want to run them manually.
Your product runs on more than Crazy Ant Labs
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services