EasyCron incident

Cron job triggerring issue

Major Resolved View vendor source →

EasyCron experienced a major incident on July 1, 2022 affecting Executors, lasting 15m. The incident has been resolved; the full update timeline is below.

Started
Jul 01, 2022, 02:57 PM UTC
Resolved
Jul 01, 2022, 03:12 PM UTC
Duration
15m
Detected by Pingoru
Jul 01, 2022, 02:57 PM UTC

Affected components

Executors

Update timeline

  1. investigating Jul 01, 2022, 02:57 PM UTC

    We're having problem triggering problem currently.

  2. identified Jul 01, 2022, 03:03 PM UTC

    multiple triggering processes locked (could be due to temporary network issue) cause freezing processing of cron job triggering. After cancelling frozen processes, and disabling HA triggering module, server is working again.

  3. monitoring Jul 01, 2022, 03:03 PM UTC

    We're monitoring the system now.

  4. resolved Jul 01, 2022, 03:12 PM UTC

    The triggering process is working good now, we will work on a fix to handle network issue when using distributed lock in the HA triggering module.