Searchspring incident

Large Indexing Queue

Minor Resolved View vendor source →

Searchspring experienced a minor incident on July 22, 2022 affecting Indexing, lasting 34m. The incident has been resolved; the full update timeline is below.

Started
Jul 22, 2022, 01:20 PM UTC
Resolved
Jul 22, 2022, 01:55 PM UTC
Duration
34m
Detected by Pingoru
Jul 22, 2022, 01:20 PM UTC

Affected components

Indexing

Update timeline

  1. investigating Jul 22, 2022, 01:20 PM UTC

    We're seeing a much larger than normal indexing queue, we're investigating to determine the cause of the increased queue size. This is resulting in longer than normal indexing times.

  2. identified Jul 22, 2022, 01:39 PM UTC

    It appears that there were a small subset clients whose indexing jobs are taking much longer than normal. This in turn created a larger than normal backlog of jobs which increased indexing times overall. It does appear with those jobs completed that indexing times are beginning to return to normal.

  3. monitoring Jul 22, 2022, 01:39 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Jul 22, 2022, 01:42 PM UTC

    The previous update was sent in error and was a default message. There was no fix that needed to be applied, the indexing queue times are continuing to decrease. We are still exploring the root cause of the higher than normal indexing cause for a small subset of customers.

  5. resolved Jul 22, 2022, 01:55 PM UTC

    Indexing times are returning to normal. We will continue to monitor.