Searchspring incident

Increased Indexing Error Rates

Minor Resolved View vendor source →

Searchspring experienced a minor incident on August 23, 2022 affecting Indexing, lasting 48m. The incident has been resolved; the full update timeline is below.

Started
Aug 23, 2022, 02:51 PM UTC
Resolved
Aug 23, 2022, 03:40 PM UTC
Duration
48m
Detected by Pingoru
Aug 23, 2022, 02:51 PM UTC

Affected components

Indexing

Update timeline

  1. investigating Aug 23, 2022, 02:51 PM UTC

    We are currently seeing higher than normal indexing error rates for a subset of our customers.

  2. investigating Aug 23, 2022, 03:10 PM UTC

    We are still investigating this issue. The issue is isolated to a set of our west coast US servers so we have directed all US traffic for these clients to our east coast servers to avoid the possibility of this issue affecting our Search APIs. Clients on this set of servers will continue to see indexing errors.

  3. monitoring Aug 23, 2022, 03:24 PM UTC

    The issue appears to have been a networking error which has recovered and indexing appears to be working normally. Indexing times may be elevated due to a larger than normal queue that was built up during this issue. We will continue to monitor the situation.

  4. monitoring Aug 23, 2022, 03:35 PM UTC

    We have directed traffic for Search API back to the US west coast for west coast customers and are continuing to monitor.

  5. resolved Aug 23, 2022, 03:40 PM UTC

    All services to have returned to normal. We will be doing further investigation to determine the root cause.