Sift incident

Delays in asynchronous event processing

Minor Resolved View vendor source →

Sift experienced a minor incident on April 4, 2019 affecting Console and API and 1 more component, lasting 8h 3m. The incident has been resolved; the full update timeline is below.

Started
Apr 04, 2019, 08:25 PM UTC
Resolved
Apr 05, 2019, 04:29 AM UTC
Duration
8h 3m
Detected by Pingoru
Apr 04, 2019, 08:25 PM UTC

Affected components

ConsoleAPIScoreWorkflows

Update timeline

  1. investigating Apr 04, 2019, 08:25 PM UTC

    We are currently experiencing a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.

  2. investigating Apr 04, 2019, 08:59 PM UTC

    We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.

  3. investigating Apr 04, 2019, 09:34 PM UTC

    We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.

  4. investigating Apr 04, 2019, 10:05 PM UTC

    We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.

  5. investigating Apr 04, 2019, 11:06 PM UTC

    We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 7 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.

  6. monitoring Apr 04, 2019, 11:37 PM UTC

    We are continuing to work through a small backlog of delayed events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.

  7. monitoring Apr 05, 2019, 12:09 AM UTC

    We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.

  8. monitoring Apr 05, 2019, 12:40 AM UTC

    We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.

  9. monitoring Apr 05, 2019, 01:11 AM UTC

    We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 2 hours after the event is received.

  10. monitoring Apr 05, 2019, 01:38 AM UTC

    The backlog of events have been processed and there is no more delay on asynchronous Scores and Workflows. There are delays to data in the Explore tab and we are currently working through the backlog of delayed messages.

  11. monitoring Apr 05, 2019, 04:27 AM UTC

    The backlog of events has been processed. All systems are healthy.

  12. resolved Apr 05, 2019, 04:29 AM UTC

    This incident has been resolved.