Crazy Ant Labs incident
Jobs that include a timezone setting are not running
Resolved after 29 minutes, affecting Web Interface and API Requests, on November 17, 2020.
Around 14:30 UTC, November 17, a deployment done by our engineers to add more details to our logs caused jobs with timezones not to execute due to errors. The issue was discovered after almost 3 hours and took some time to revert. In order to avoid this kind of issues in the future, we’ve added more alerts in the infrastructure that supports these jobs and added automation to our tests of jobs with timezones.
Affected components
Update timeline
- investigating Nov 17, 2020, 06:16 PM UTC
We are currently investigating this issue.
- investigating Nov 17, 2020, 06:22 PM UTC
We are continuing to investigate this issue.
- resolved Nov 17, 2020, 06:22 PM UTC
This incident has been resolved.
- postmortem Nov 19, 2020, 06:30 PM UTC
Around 14:30 UTC, November 17, a deployment done by our engineers to add more details to our logs caused jobs with timezones not to execute due to errors. The issue was discovered after almost 3 hours and took some time to revert. In order to avoid this kind of issues in the future, we’ve added more alerts in the infrastructure that supports these jobs and added automation to our tests of jobs with timezones.
Your product runs on more than Crazy Ant Labs
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services