LoyaltyLion incident

Job processing delays

Minor Resolved View vendor source →

LoyaltyLion experienced a minor incident on March 19, 2021 affecting Platform and SDK and Merchant account, lasting 1h 20m. The incident has been resolved; the full update timeline is below.

Started
Mar 19, 2021, 12:50 PM UTC
Resolved
Mar 19, 2021, 02:11 PM UTC
Duration
1h 20m
Detected by Pingoru
Mar 19, 2021, 12:50 PM UTC

Affected components

Platform and SDKMerchant account

Update timeline

  1. investigating Mar 19, 2021, 12:50 PM UTC

    We're currently investigating an issue relating to our background job processing. You may see delays in the following areas: * Customer email delivery * Launching loyalty tiers * Imports and exports Core customer-facing functionality including earning points, claiming rewards, and customer UI is unaffected.

  2. identified Mar 19, 2021, 01:17 PM UTC

    We've identified the root cause of the issue; we've triaged the problem and are working on a permanent fix.

  3. monitoring Mar 19, 2021, 01:38 PM UTC

    We've just rolled out a fix and have resumed background job processing; we'll give another update when we've cleared the backlog.

  4. resolved Mar 19, 2021, 02:11 PM UTC

    We've caught up with our backlog and all delayed jobs have now been processed.