Coveo HIPAA incident

Analytics Data Processing and Document Processing Issues

Minor Resolved View vendor source →

Coveo HIPAA experienced a minor incident on June 13, 2023 affecting Platform - Authentication Service and Indexing Pipeline - Push API and 1 more component, lasting 3h 12m. The incident has been resolved; the full update timeline is below.

Started
Jun 13, 2023, 07:16 PM UTC
Resolved
Jun 13, 2023, 10:28 PM UTC
Duration
3h 12m
Detected by Pingoru
Jun 13, 2023, 07:16 PM UTC

Affected components

Platform - Authentication ServiceIndexing Pipeline - Push APIIndexing Pipeline - CrawlingAnalytics - Analytics Write APIAnalytics - Analytics Read API

Update timeline

  1. investigating Jun 13, 2023, 07:16 PM UTC

    We are currently experiencing a data processing issue within the US region that is preventing events collected since 02:50 EST today from being processed accordingly. This impacts mostly reporting and analytics features. We are also seeing elevated error rates on the Push API, preventing documents and identities from being sent to Coveo properly. If you need help or to get in touch with us, please visit our Help Portal

  2. identified Jun 13, 2023, 07:52 PM UTC

    We've identified that the issue is also impacting crawling operations, so your documents may not be up to date. The Log Browser is currently inaccessible as well. Personalization features might be affected by the outage.

  3. identified Jun 13, 2023, 08:22 PM UTC

    We've identified that logging in by email, or by following links in invitation emails to organizations is also impacted.

  4. monitoring Jun 13, 2023, 08:59 PM UTC

    We are starting to see recovery across affected services, and continue to monitor the situation closely. However, some services have built up a backlog of events that will take some time to process.

  5. resolved Jun 13, 2023, 10:28 PM UTC

    All services have recovered, and the backlogs have been processed. Please report any issues through our Help Portal.