CrystalCommerce incident

Job Delays - 12/9/21

Minor Resolved View vendor source →

CrystalCommerce experienced a minor incident on December 9, 2021 affecting Admin and POS and 1 more component, lasting 3h 23m. The incident has been resolved; the full update timeline is below.

Started
Dec 09, 2021, 04:21 PM UTC
Resolved
Dec 09, 2021, 07:45 PM UTC
Duration
3h 23m
Detected by Pingoru
Dec 09, 2021, 04:21 PM UTC

Affected components

AdminPOSBatch Update

Update timeline

  1. identified Dec 09, 2021, 04:21 PM UTC

    We are seeing some delays with jobs processing this morning 12/9/21 , causing interruptions to various functions in the admin. Our engineering team is working to speed things up.

  2. monitoring Dec 09, 2021, 05:35 PM UTC

    Our developers have deployed a fix for the stalled background jobs queue, and we are seeing them being processed. Batch updates, orders status changes, and other background processes are delayed, but they are running, and will take some time to complete. We anticipate that batch updates will take the longest to catch up.

  3. monitoring Dec 09, 2021, 06:51 PM UTC

    All queues are clear except for batch updates, which has a little further to go before it's all caught up.

  4. resolved Dec 09, 2021, 07:45 PM UTC

    All Queues are caught up.