Retreaver incident

Call log and dashboards were down

Minor Resolved View vendor source →

Retreaver experienced a minor incident on November 6, 2018 affecting Site/Account Portal, lasting 4h 46m. The incident has been resolved; the full update timeline is below.

Started
Nov 06, 2018, 12:31 PM UTC
Resolved
Nov 06, 2018, 05:17 PM UTC
Duration
4h 46m
Detected by Pingoru
Nov 06, 2018, 12:31 PM UTC

Affected components

Site/Account Portal

Update timeline

  1. identified Nov 06, 2018, 12:31 PM UTC

    We're working on a fix. Calls continue to be processed as normal. Services will resume as normal shortly.

  2. identified Nov 06, 2018, 12:55 PM UTC

    We have restored our search servers. Phone numbers have been fully synced. Live call data is up to date, but the call log is currently being backfilled. Note that historical call data will remain out of sync for approximately 2 hours as we reindex the calls in reverse chronological order. We apologize for the inconvenience this morning, this issue will be updated again when calls have been fully reindexed.

  3. monitoring Nov 06, 2018, 01:07 PM UTC

    We have restored our search servers. Phone numbers have been fully synced. Live call data is up to date, but the call log is currently being backfilled. Note that historical call data will remain out of sync for approximately 2 hours as we reindex the calls in reverse chronological order. We apologize for the inconvenience this morning, this issue will be updated again when calls have been fully reindexed.

  4. monitoring Nov 06, 2018, 03:49 PM UTC

    Historical call data continues to be backfilled (calls prior to April 10 2018). We expect the process to complete around noon EST. All other functionality continues to operate as normal, and call flow was not affected during this incident. We apologize for the inconvenience this morning, this issue will be updated again once calls have been fully synced to our search servers.

  5. resolved Nov 06, 2018, 05:17 PM UTC

    All calls have been reindexed and the service has returned to normal.