LoyaltyLion incident

Job queue partial outage

Major Resolved View vendor source →

LoyaltyLion experienced a major incident on March 4, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
Mar 04, 2024, 05:21 PM UTC
Resolved
Mar 02, 2024, 01:30 PM UTC
Duration
Detected by Pingoru
Mar 04, 2024, 05:21 PM UTC

Update timeline

  1. resolved Mar 04, 2024, 05:21 PM UTC

    We observed a performance degradation in one of our job queueing subsystems. As part of recovery to normal operations, we - paused processing on customer tier evaluation between 2024-03-02T13:30 UTC and 2024-03-02T19:00 UTC - paused processing on outbound webhook delivery between 2024-03-02T14:30 UTC and 2024-03-03T14:30 UTC - implemented controls on unusual traffic from Shopify Tier assignment and webhook delivery are now functioning as normal. We plan to gradually deliver the webhooks from the paused period over the next few days.