Sift incident

Delays in asynchronous event processing

Minor Resolved View vendor source →

Sift experienced a minor incident on April 2, 2019 affecting API, lasting 1d 6h. The incident has been resolved; the full update timeline is below.

Started
Apr 02, 2019, 07:57 AM UTC
Resolved
Apr 03, 2019, 02:45 PM UTC
Duration
1d 6h
Detected by Pingoru
Apr 02, 2019, 07:57 AM UTC

Affected components

API

Update timeline

  1. monitoring Apr 02, 2019, 07:58 AM UTC

    We are currently experiencing a processing delay for events received through the Event API. Updated scores, data in Explore, and workflows may be delayed. We have implemented a fix and are monitoring. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.

  2. monitoring Apr 02, 2019, 11:14 AM UTC

    We are adding additional capacity to address the processing delay.

  3. monitoring Apr 02, 2019, 04:33 PM UTC

    We continue to actively monitoring the delay in asynchronous processing.

  4. monitoring Apr 02, 2019, 05:06 PM UTC

    We continue to actively monitor the delay in asynchronous processing.

  5. monitoring Apr 02, 2019, 05:40 PM UTC

    We continue to actively monitor the delay in asynchronous processing.

  6. monitoring Apr 02, 2019, 06:12 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  7. monitoring Apr 02, 2019, 06:43 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  8. monitoring Apr 02, 2019, 07:15 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  9. monitoring Apr 02, 2019, 07:46 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  10. monitoring Apr 02, 2019, 08:17 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  11. monitoring Apr 02, 2019, 08:46 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  12. monitoring Apr 02, 2019, 09:22 PM UTC

    We are continuing to actively monitor the delay in asynchronous processing.

  13. identified Apr 02, 2019, 09:57 PM UTC

    The issue with delays to asynchronous processing has been identified and a fix is being deployed.

  14. identified Apr 02, 2019, 10:42 PM UTC

    The issue with delays to asynchronous processing has been identified and a fix has been deployed. We will continue to process delayed messages and provide updates.

  15. identified Apr 02, 2019, 11:21 PM UTC

    The issue with delays to asynchronous processing has been identified and a fix has been deployed. We are continuing to process delayed messages.

  16. identified Apr 02, 2019, 11:53 PM UTC

    The issue with delays to asynchronous processing has been identified and a fix has been deployed. We are continuing to work through a backlog of delayed messages.

  17. identified Apr 03, 2019, 12:24 AM UTC

    We are continuing to work through a backlog of messages and expect to resolve delays in 8 hours. We'll update this estimate if this changes.

  18. monitoring Apr 03, 2019, 09:39 AM UTC

    The backlog of events have been processed and there is no more delay on asynchronous processing.

  19. monitoring Apr 03, 2019, 11:39 AM UTC

    The backlog of events have been processed and there is no more delay on asynchronous Scores and Workflows. There are delays to data in the Explore tab and we are currently working through the backlog of delayed messages.

  20. monitoring Apr 03, 2019, 12:15 PM UTC

    The backlog of events have been processed and there is no more delay on asynchronous processing.

  21. resolved Apr 03, 2019, 02:45 PM UTC

    All systems are healthy.