Spruce Health incident

Stale search results when searching for contacts, conversations or messages

Minor Resolved View vendor source →

Spruce Health experienced a minor incident on October 24, 2024 affecting Web App and Mobile Apps, lasting 21h 31m. The incident has been resolved; the full update timeline is below.

Started
Oct 24, 2024, 07:00 PM UTC
Resolved
Oct 25, 2024, 04:32 PM UTC
Duration
21h 31m
Detected by Pingoru
Oct 24, 2024, 07:00 PM UTC

Affected components

Web AppMobile Apps

Update timeline

  1. investigating Oct 24, 2024, 10:44 PM UTC

    We're seeing elevated latencies in indexing contacts, conversations and messages for the purpose of searching. This is resulting in potentially stale search results being returned when searching for contacts, conversations and messages or even when loading contact lists. This issue started around 12pm PT / 3pm ET and is ongoing. We're actively investigating the issue and will post updates here as soon as we have them. There is no impact to loading the Spruce inbox, making/receiving calls, exchanging SMS, email or fax on Spruce.

  2. identified Oct 25, 2024, 12:47 AM UTC

    We have identified the issue and are working through a plan to fully resolve the issue. The latency for updates in the system to show up in the search results is ~15 minutes. We're working to see how we can improve this and fully resolve the issue.

  3. resolved Oct 25, 2024, 04:32 PM UTC

    Search latency is back to a normal state as of 2am PT / 5 am ET October 25, 2024. We have a good understanding of what happened here and are looking into ways we can reduce the likelihood of a similar situation in the future. Customer impact was from ~12pm PT / 3pm ET on October 24, 2024 to 2am PT / 5 am ET where search results were stale for conversation, contact and contact list querying. There was no impact to the Spruce inbox, exchanging of calls, SMS, secure messaging, email, video calls or fax on the platform.