Platform Issues

Update timeline

1. investigating Dec 22, 2021, 02:14 PM UTC

   We are investigating availability issues for a significant portion of our platform due to issues with the AWS and Heroku platforms.

2. identified Dec 22, 2021, 02:27 PM UTC

   The issue has been identified with Heroku's US region availability issues. Customers are not able to login to our add-ons using Heroku's SSO login mechanism an Cron To Go jobs are failing intermittently due to Heroku API availability issues.

3. identified Dec 22, 2021, 02:33 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Dec 22, 2021, 03:12 PM UTC

   Heroku and AWS identified the issue and are actively working on a fix. We and are seeing recovery for the majority of our services but we still see a backlog of Cron To Go jobs that are slowly getting processed.

5. monitoring Dec 22, 2021, 06:23 PM UTC

   All Cron To Go jobs back log has caught up and we see full recovery in all systems. We keep monitoring the situation to ensure the issue has been resolved.

6. resolved Dec 24, 2021, 02:06 PM UTC

   This incident has been resolved.

7. postmortem Dec 24, 2021, 02:08 PM UTC

   The issues on AWS - that were caused due to electricity issues in one of their US data centers - affected our products as well as Heroku. In Cron To Go, new issues arised after the AWS issues were fixed since Cron To Go was executing jobs but still received errors from Heroku APIs and retried execution. This caused the job queue to bloat up more than the workers' capacity and caused jobs to run out of schedule \(due to retries\).