Fat Zebra incident

Performance of search facilities in merchant and reseller dashboards

Major Resolved View vendor source →

Fat Zebra experienced a major incident on April 17, 2020 affecting Dashboard, lasting 12h 49m. The incident has been resolved; the full update timeline is below.

Started
Apr 17, 2020, 01:00 AM UTC
Resolved
Apr 17, 2020, 01:50 PM UTC
Duration
12h 49m
Detected by Pingoru
Apr 17, 2020, 01:00 AM UTC

Affected components

Dashboard

Update timeline

  1. investigating Apr 17, 2020, 01:00 AM UTC

    Performance of search facilities in merchant and reseller dashboards is performing slowly and is being investigated.

  2. investigating Apr 17, 2020, 02:21 AM UTC

    The issue is ongoing and we have engaged Amazon Web Services to investigate the issue.

  3. investigating Apr 17, 2020, 02:40 AM UTC

    Currently the delivery of webhooks has been delayed.

  4. investigating Apr 17, 2020, 03:29 AM UTC

    Webhooks have been restored and we are currently working on restoring search.

  5. investigating Apr 17, 2020, 03:49 AM UTC

    Search has been partially restored but is limited to newer records. Older records are being progressively restored.

  6. investigating Apr 17, 2020, 04:15 AM UTC

    We are continuing to investigate this issue.

  7. monitoring Apr 17, 2020, 07:11 AM UTC

    The majority of indexes are up to date. There still remains a delay with older records for purchases, however we are working on prioritising these and will provide an update shortly.

  8. resolved Apr 17, 2020, 01:50 PM UTC

    Indexing of recent records is now complete, and indexing of older records is continuing as a background task. We will continue to monitor these indexes until the backlog is completed. If merchants are still experiencing issues searching please contact our support team for further assistance. We apologise for the inconvenience this has cause to our merchants and their customers.