Coveo HIPAA incident

Analytics Service - Write & Indexing issues

Major Resolved View vendor source →

Coveo HIPAA experienced a major incident on November 25, 2020 affecting Analytics - Analytics Write API, lasting 1d 4h. The incident has been resolved; the full update timeline is below.

Started
Nov 25, 2020, 01:36 PM UTC
Resolved
Nov 26, 2020, 06:16 PM UTC
Duration
1d 4h
Detected by Pingoru
Nov 25, 2020, 01:36 PM UTC

Affected components

Analytics - Analytics Write API

Update timeline

  1. identified Nov 25, 2020, 01:36 PM UTC

    We've identified an issue affecting Usage Analytics Write APIs. A number of events sent since 8:14am Eastern time might be irrecoverable. We've identified the problem being with our infrastructure provider and we're in communication with them while they investigate. If you need help or to get in touch with us, please visit our Help Portal

  2. identified Nov 25, 2020, 03:19 PM UTC

    The issue is also impacting indexing so your sources content might not be up to date. We're still in close communication with our provider.

  3. identified Nov 25, 2020, 06:26 PM UTC

    Our infrastructure provider is still working on getting this issue solved but has no ETA to share. You can follow their progression on AWS Service Health Dashboard. In parallel, we're working on ways to mitigate the outage and will update you with our progress.

  4. identified Nov 25, 2020, 08:08 PM UTC

    Indexing is fixed and the backlog has been processed. Some longer sources refreshes are still ongoing. This service is back to normal and we are actively monitoring it.

  5. monitoring Nov 25, 2020, 09:52 PM UTC

    Our provider is gradually recovering and our workarounds allowed us to stabilize the Analytics events writes. We are actively monitoring the service.

  6. monitoring Nov 26, 2020, 03:37 AM UTC

    Our provider has mostly recovered and our services are still up and functional since our last update. We will keep our mitigation measures until tomorrow morning (Eastern time), reassess the situation then and update this incident. We are monitoring the situation and are in close communications with our provider.

  7. monitoring Nov 26, 2020, 03:10 PM UTC

    The services are still up and running since the last update. We’re catching up with our ML Usage Analytics events backlog and expect to be done within ~3 hours. We’ll post a final update by then.

  8. resolved Nov 26, 2020, 06:16 PM UTC

    Everything is up to date and the services are still functional. Please report any issues through our Help Portal.