Coveo HIPAA incident

Various services impacted by major AWS Issues

Major Resolved View vendor source →

Coveo HIPAA experienced a major incident on December 7, 2021 affecting Platform - Administration Console and Indexing Pipeline - Source Service and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 03:54 PM UTC
Resolved
Dec 08, 2021, 04:21 PM UTC
Duration
1d
Detected by Pingoru
Dec 07, 2021, 03:54 PM UTC

Affected components

Platform - Administration ConsoleIndexing Pipeline - Source ServiceIndexing Pipeline - Push APIIndexing Pipeline - Crawling ModuleAnalytics - Analytics Write APICoveo ML - Models Generator

Update timeline

  1. identified Dec 07, 2021, 03:54 PM UTC

    Some of our services are impacted by an AWS issue - Crawling & Push API is delayed - Analytics events write is delayed - Models Building is delayed Search & models querying are not impacted. We're in talk with our provider and will post regular updates. If you need help or to get in touch with us, please visit our Help Portal

  2. identified Dec 07, 2021, 04:15 PM UTC

    We're still in talk with our Infrastructure Provider but have no ETA to share as of now.

  3. identified Dec 07, 2021, 05:00 PM UTC

    Our Infrastructure Provider has identified the issue and is actively working on addressing it but has no ETA to share at the moment.

  4. identified Dec 07, 2021, 05:34 PM UTC

    We've updated the status of our components. Again, still in talk with AWS but they have no ETA to share yet.

  5. identified Dec 07, 2021, 05:35 PM UTC

    We've updated the status of our components.

  6. identified Dec 07, 2021, 07:19 PM UTC

    We're still in talk with AWS but as of now, they still don't have any ETA to share.

  7. identified Dec 07, 2021, 09:08 PM UTC

    We're still working on mitigation for our service which are still impacted.

  8. identified Dec 07, 2021, 10:48 PM UTC

    We're seeing signs of recovery from our infrastructure provider and we are closely monitoring the impacted services.

  9. identified Dec 07, 2021, 11:38 PM UTC

    We've identified a new issue preventing the Analytics write service from accepting new events and are working on mitigating the impact.

  10. identified Dec 07, 2021, 11:57 PM UTC

    We've mitigated the issue impacting the Analytics write service. It is now operating normally again. We continue to monitor all services closely.

  11. identified Dec 08, 2021, 12:25 AM UTC

    We are still in the phase of fully recovering our services but we wanted to provide you with an update. A more detailed and thorough explanation will be provided through the standard RCA process when it is completed. At 10:36am EST on December 7, 2021, the Coveo team became aware of issues with a third-party cloud services provider that had impacted multiple Coveo services. The provider has communicated that they have taken mitigation actions that show significant recovery in the impacted region. They expect to continue to see improved performance, but do not have an ETA for full recovery at this time. We are managing the situation and we will keep you informed as it evolves. Thank you

  12. monitoring Dec 08, 2021, 02:50 AM UTC

    The issues have been either resolved or mitigated and all our services are operating normally. We will continue to monitor the services. Please report any issues through our Help Portal.

  13. resolved Dec 08, 2021, 04:21 PM UTC

    The services continue to be fully operational since our last update and we’re closing this incident. As stated earlier, a more detailed explanation will be provided through the standard RCA process when it is completed. Please report any issues through our Help Portal. Thank you