DANAconnect incident

We are currently experiencing an outage affecting Conversations Processing. Our Engineering Team is aware and working to resolve.

Critical Resolved View vendor source →

DANAconnect experienced a critical incident on February 23, 2023 affecting Email platform and Director - Workflow Orchestration and 1 more component, lasting 2h 39m. The incident has been resolved; the full update timeline is below.

Started
Feb 23, 2023, 04:59 PM UTC
Resolved
Feb 23, 2023, 07:38 PM UTC
Duration
2h 39m
Detected by Pingoru
Feb 23, 2023, 04:59 PM UTC

Affected components

Email platformDirector - Workflow OrchestrationWebhooks (API Request Node)Conversation APISMTP Service

Update timeline

  1. investigating Feb 23, 2023, 04:59 PM UTC

    We are currently investigating this issue.

  2. investigating Feb 23, 2023, 05:20 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Feb 23, 2023, 05:55 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Feb 23, 2023, 06:13 PM UTC

    The outage was originated from the Amazon MQ queues component. Our clustered message orchestrators were connected to the queuing system, not logging errors, but messages were not being processed. When the queuing system was restored, the messages continued to be processed satisfactorily.

  5. resolved Feb 23, 2023, 07:38 PM UTC

    This incident has been resolved.