DANAconnect Outage History

Started Sep 30, 2026, 05:21 PM UTC· resolved Sep 30, 2026, 06:42 PM UTC

1. investigating Sep 30, 2026, 05:21 PM UTC

   We are currently experiencing delays in shipping processing and dispatch due to a high volume of traffic on our services. Our team is actively working to resolve the issue and restore normal operations as quickly as possible.

2. identified Sep 30, 2026, 06:03 PM UTC

   We are pleased to inform you that our shipping and dispatch processing services have stabilized and returned to normal operations. Our team continues to actively monitor the situation to ensure seamless performance

3. monitoring Sep 30, 2026, 06:03 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 30, 2026, 06:42 PM UTC

   Procesamiento de Envíos y Despachos Servicio afectado: Proceso de Envíos y Despacho (Shipping & Dispatch Processing Services) Estado: Resuelto / Estabilizado 1. Resumen Los servicios de procesamiento de envíos y despachos experimentaron una degradación de rendimiento y retrasos operativos debido a un alto volumen de transacciones de actualización sobre la base de datos. El incidente comenzó a manifestar un incremento sostenido de latencia y procesamiento alrededor de las 12:21 EDT, alcanzando un pico crítico inicial entre las 12:22 y las 12:24 EDT, y manteniendo un nivel elevado de carga y saturación constante desde las 12:38 EDT hasta las 13:38 EDT. Tras la aplicación de una corrección de emergencia en la lógica de actualización, el servicio retornó a parámetros normales operacionales a las 13:43 EDT, confirmándose la resolución y estabilización definitiva a las 14:03 EDT. 2. Cronología del Incidente (Timeline) 12:15 EDT - 12:20 EDT: Inicio del incremento gradual en las solicitudes de procesamiento. 12:21 EDT: Caída momentánea seguida de un pico pronunciado de solicitudes/latencia a las 12:22 - 12:24 EDT. 12:25 EDT - 12:35 EDT: Fluctuación del volumen por encima del umbral promedio normal (línea punteada). 12:38 EDT: Comienza el periodo sostenido de alta saturación del servicio debido a la contención en la base de datos. 13:00 EDT - 13:30 EDT: Mantenimiento prolongado del pico de saturación al límite operacional del sistema. 13:38 EDT: Despliegue de la solución y comienzo del descenso progresivo de la carga acumulada. 13:40 EDT: Caída drástica del encolamiento tras la ejecución del nuevo proceso unificado de actualización. 13:43 EDT: Las métricas de rendimiento y tiempos de respuesta vuelven a los niveles baseline normales. 14:03 EDT: Publicación oficial de la resolución del incidente tras verificar la estabilidad continua del sistema ("Identified - Services stabilized and returned to normal operations"). 3. Análisis de Impacto Duración total del incidente: ~1 hora y 22 minutos (degradación registrada desde las 12:21 hasta las 13:43 EDT). Duración de la degradación crítica: ~1 hora (12:38 EDT – 13:38 EDT). Servicios afectados: Módulo de procesamiento de envíos,y cola de peticiones asociadas. 4. Causa Raíz El incidente fue provocado por un elevado incremento en el volumen de actualizaciones (updates) sobre la tabla de despachos de la base de datos. La arquitectura ejecutaba estas actualizaciones de forma escalonada por bloques mediante múltiples procesos individuales y simultáneos. Al aumentar la carga, la ejecución concurrente sobre la misma tabla generó un bloqueo de recursos (table/row locking) y una alta contención en la base de datos, derivando en un cuello de botella que saturó el procesamiento de envíos entre las 12:38 EDT y las 13:38 EDT. 5. Acciones de Mitigación y Resolución Corrección de Emergencia (Hotfix): El equipo DevOps implementó un ajuste de emergencia para consolidar y aplicar las actualizaciones de manera masiva a través de un único proceso optimizado, eliminando la contención por múltiples procesos en paralelo. Descongestión del Sistema: Al reducirse los bloqueos en la tabla, el backlog acumulado se procesó rápidamente. La carga cayó drásticamente a las 13:40 EDT, logrando la estabilización completa del servicio a las 13:43 EDT y la confirmación final de normalidad a las 14:03 EDT.

Read the full incident report →

Started Jul 03, 2026, 11:30 PM UTC· resolved Jul 04, 2026, 01:30 AM UTC

1. investigating Jul 03, 2026, 11:30 PM UTC

   An issue affecting the message processing layer caused intermittent delays in dispatch operations during periods of elevated workload. We apologize for any inconveniences this may cause. Our personnel are currently working to resolve the situation and restore normal operations. We will keep you updated on our progress.

2. identified Jul 04, 2026, 12:38 AM UTC

   We have found the cause of the system saturation, and the team has successfully stabilized the service. All dispatch operations are now catching up and stabilizing. Thank you for your patience while we resolved this.

3. monitoring Jul 04, 2026, 12:45 AM UTC

   The issue affecting the message processing layer has been fully resolved. The team successfully addressed the root cause, the service is now completely stabilized, and all dispatch operations have returned to normal parameters. We are now closing this incident and transitioning the system to monitoring mode to ensure continued stability. Thank you for your patience and cooperation.

4. resolved Jul 04, 2026, 01:30 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 01, 2026, 03:26 AM UTC· resolved Jul 01, 2026, 11:34 AM UTC

1. investigating Jul 01, 2026, 03:26 AM UTC

   We are currently experiencing service disruptions affecting access to the User Portal. Our team is already investigating the root cause of the issue to restore service to normal as soon as possible. We apologize for any inconvenience this may cause and will keep this page updated as more information becomes available.

2. resolved Jul 01, 2026, 11:34 AM UTC

   This incident has been resolved.

3. postmortem Jul 01, 2026, 07:27 PM UTC

   We have completed our investigation and identified the root cause of the incident. The issue was traced to **database session management** within our infrastructure, which under specific conditions **resulted in intermittent interruptions during the authentication and validation process in the access portal**. During the investigation, our engineering team conducted a comprehensive analysis and evaluated multiple mitigation approaches to restore normal operation while preserving application consistency and data integrity. Following these efforts, the environment was stabilized, **service has been fully restored**, and all affected components are operating normally. We have also identified the underlying factors that contributed to this behavior and are implementing permanent improvements to strengthen our database connectivity layer, enhance overall platform resilience, and reduce the likelihood of similar issues occurring in the future. We sincerely appreciate your patience and understanding while we worked to resolve this issue. ‌ DANAconnect Incident Team

Read the full incident report →

Started Jan 22, 2026, 03:01 PM UTC· resolved Jan 22, 2026, 05:00 PM UTC

1. investigating Jan 22, 2026, 03:01 PM UTC

   We are experiencing issues displaying real time reporting. The source of the problem is a reference to a service on w3.org. The DevOps team is working to enable the display. All events and processing are being logged.

2. identified Jan 22, 2026, 03:03 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jan 22, 2026, 04:19 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 22, 2026, 05:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 02, 2025, 03:47 PM UTC· resolved Dec 02, 2025, 05:45 PM UTC

1. investigating Dec 02, 2025, 03:47 PM UTC

   Estamos observando una degradación en el despacho de SMS y estamos investigando las posibles causas en coordinación con las operadoras correspondientes. Nuestro equipo continúa trabajando para normalizar el servicio. Compartiremos actualizaciones en cuanto tengamos más información

2. identified Dec 02, 2025, 04:09 PM UTC

   Causa Identificada en el Servicio DNS Nos han indicado que la degradación en el envío de SMS está relacionada con una afectación en el servicio DNS del centro de datos utilizado por las operadoras. Nos mantenemos en monitoreo continuo del comportamiento del servicio mientras los equipos técnicos correspondientes realizan las acciones necesarias para la recuperación. Compartiremos nuevas actualizaciones cuando haya información adicional.

3. monitoring Dec 02, 2025, 05:42 PM UTC

   La afectación relacionada con el envío de SMS ha sido solventada por las operadoras. Hemos pasado a modo de monitoreo para confirmar la estabilidad del servicio y asegurar que el desempeño se mantenga dentro de los parámetros habituales. Informaremos cualquier novedad relevante en caso de ser necesario.

4. resolved Dec 02, 2025, 05:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 01, 2025, 10:04 AM UTC· resolved Dec 01, 2025, 08:22 PM UTC

1. investigating Dec 01, 2025, 10:04 AM UTC

   We have detected an intermittent issue affecting the sending and delivery of emails from the platform. This may cause delays or occasional failures in message reception. Our team is already working closely with the service provider to identify the root cause and apply the necessary fixes as soon as possible. We appreciate your patience while we move forward with the resolution. We will share updates as soon as we have confirmed information.

2. identified Dec 01, 2025, 11:02 AM UTC

   DNS Service Degradation. We are currently observing disruptions in our email dispatch and delivery processes caused by an ongoing incident affecting our provider’s DNS services. This issue may lead to delays or intermittent failures when attempting to send or receive messages through the platform. Our engineering team is actively working with the provider to assess the impact and restore normal service as quickly as possible. Further updates will be posted as soon as additional verified information becomes available.

3. identified Dec 01, 2025, 12:17 PM UTC

   The provider continues investigating the incident related to the degradation of its DNS services. No new elements have been identified so far that provide further clarity, but analysis efforts remain ongoing to determine the origin of the observed behavior. Our team remains in constant communication with the provider to monitor progress and gather verified information as soon as it becomes available. We will publish a new update as soon as we have confirmed developments.

4. identified Dec 01, 2025, 01:20 PM UTC

   The provider continues working to resolve the incident affecting its DNS services and has indicated that technical tasks remain in progress. At this time, we do not have an estimated resolution time, but we will share it as soon as the provider makes that information available. Our team remains in constant contact with them to monitor progress and gather verified data as soon as it becomes available. We will publish a new update as soon as we have confirmed developments.

5. identified Dec 01, 2025, 02:21 PM UTC

   The provider continues working to resolve the incident affecting its DNS services and has indicated that technical tasks remain in progress. At this time, we do not have an estimated resolution time, but we will share it as soon as the provider makes that information available. Our team remains in constant contact with them to monitor progress and gather verified data as soon as it becomes available. We will publish a new update as soon as we have confirmed developments.

6. identified Dec 01, 2025, 03:12 PM UTC

   Update : The provider has identified the root cause of the incident affecting its DNS services. The issue originates from a faulty PDU impacting critical network-layer equipment and communications between the provider’s data centers. Their technical team is currently reviewing and replacing the affected component. We remain in direct contact with them to obtain verified information regarding progress and next steps. We will publish a new update as soon as we have confirmed details from the provider.

7. identified Dec 01, 2025, 05:25 PM UTC

   The provider continues carrying out recovery work on the affected infrastructure. Their technical team is performing the necessary adjustments and validations to fully restore the impacted services. We remain in direct communication with the provider to obtain verified progress updates. We will share a new status report as soon as we receive confirmed information.

8. monitoring Dec 01, 2025, 05:48 PM UTC

   The service has been restored and is currently operating normally. Systems are responding as expected following the actions taken by the provider on its infrastructure. We will maintain continuous monitoring until we receive the provider’s final confirmation certifying full service stability. We will share any relevant updates as soon as they are verified.

9. resolved Dec 01, 2025, 08:22 PM UTC

   This incident has been resolved.

10. postmortem Dec 01, 2025, 08:22 PM UTC

    **Incident summary: Intermittent email delivery issues** On December 1st, an incident occurred that caused intermittent failures exclusively in the platform’s email sending service. All other services operated normally and without interruptions. The issue was linked to a degradation in the provider’s DNS services. According to validated information, the incident was caused by a faulty PDU that affected critical network equipment and the communication between the provider’s data centers. This led to DNS instability and intermittently impacted email dispatch. During the event, the provider carried out diagnostic tasks, adjustments, and the replacement of the affected component, while we maintained direct communication to obtain verified updates and monitor the service status. At 12:48 EST, the provider completed the recovery actions, and the email sending service returned to normal operation. Although the service stabilized, we continued monitoring until receiving the provider’s final confirmation of full stability. The total duration of the impact was **7 hours and 44 minutes**.

Read the full incident report →

Started Nov 25, 2025, 10:55 PM UTC· resolved Nov 25, 2025, 11:58 PM UTC

1. investigating Nov 25, 2025, 10:55 PM UTC

   We are experiencing delays in the delivery of outbound messages. We will provide updates as we work towards a resolution.

2. identified Nov 25, 2025, 11:01 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Nov 25, 2025, 11:11 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Nov 25, 2025, 11:24 PM UTC

   During a period of exceptionally high message-processing volume, the DANAconnect platform began receiving an elevated number of error logs generated by one of the internal background processes. These errors created congestion within the processing pipeline, which in turn caused delays in the dispatching of outbound messages. Our engineering team promptly investigated the issue and identified the specific massive process responsible for triggering the errors under heavy load. The process was corrected, and normal message-processing performance was restored. We continue to monitor the platform closely to ensure stable and timely message delivery. Thank you for your understanding.

5. resolved Nov 25, 2025, 11:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 03, 2025, 02:14 PM UTC· resolved Nov 03, 2025, 02:32 PM UTC

1. investigating Nov 03, 2025, 02:14 PM UTC

   Interrupción Parcial del Servicio de Chat Crisp Estamos experimentando una interrupción parcial en el servicio de chat en vivo de Crisp. Algunos usuarios pueden tener problemas para enviar o recibir mensajes.

2. identified Nov 03, 2025, 02:17 PM UTC

   Hemos recibido una notificación de nuestro proveedor del servicio de chat, Crisp, confirmando que están experimentando fallas serias y generalizadas en varios de sus componentes principales. Su equipo de ingeniería está trabajando activamente en la resolución. Impacto: Esta situación está afectando de manera intermitente o total el servicio de chat en vivo y la recepción de mensajes. Nuestras Acciones: Estamos monitoreando de cerca las actualizaciones de Crisp y confirmando el restablecimiento de nuestro servicio tan pronto como el incidente sea resuelto por su parte. Lamentamos profundamente las molestias causadas por esta situación fuera de nuestro control. Agradecemos su comprensión.

3. resolved Nov 03, 2025, 02:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 01:31 PM UTC· resolved Oct 21, 2025, 12:15 AM UTC

1. monitoring Oct 20, 2025, 01:31 PM UTC

   Since approximately 2:51 am EDT, some services of our cloud provider AWS were impacted. This issues that may be impacting multiple services. At this time the platform is operative but with intermittence. Here is the communication log from AWS: [12:11 AM PDT] We are investigating increased error rates and latencies for multiple AWS services in the US-EAST-1 Region. We will provide another update in the next 30-45 minutes. [12:51 AM PDT] We can confirm increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause. We will provide an update in 45 minutes, or sooner if we have additional information to share. [01:26 AM PDT] We can confirm significant error rates for requests made to the DynamoDB endpoint in the US-EAST-1 Region. This issue also affects other AWS Services in the US-EAST-1 Region as well. During this time, customers may be unable to create or update Support Cases. Engineers were immediately engaged and are actively working on both mitigating the issue, and fully understanding the root cause. We will continue to provide updates as we have more information to share, or by 2:00 AM. [02:01 AM PDT] We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery. This issue also affects other AWS Services in the US-EAST-1 Region. Global services or features that rely on US-EAST-1 endpoints such as IAM updates and DynamoDB Global tables may also be experiencing issues. During this time, customers may be unable to create or update Support Cases. We recommend customers continue to retry any failed requests. We will continue to provide updates as we have more information to share, or by 2:45 AM. [02:22 AM PDT] We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM. [02:27 AM PDT] We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information. [03:03 AM PDT] We continue to observe recovery across most of the affected AWS Services. We can confirm global services and features that rely on US-EAST-1 have also recovered. We continue to work towards full resolution and will provide updates as we have more information to share. [03:35 AM PDT] The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2 instances (or services that launch EC2 instances such as ECS) in the US-EAST-1 Region are still experiencing increased error rates. We continue to work toward full resolution. If you are still experiencing an issue resolving the DynamoDB service endpoints in US-EAST-1, we recommend flushing your DNS caches. We will provide an update by 4:15 AM, or sooner if we have additional information to share. [04:08 AM PDT] We are continuing to work towards full recovery for EC2 launch errors, which may manifest as an Insufficient Capacity Error. Additionally, we continue to work toward mitigation for elevated polling delays for Lambda, specifically for Lambda Event Source Mappings for SQS. We will provide an update by 5:00 AM PDT. [04:48 AM PDT] We continue to work to fully restore new EC2 launches in US-EAST-1. We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically. [05:10 AM PDT] We confirm that we have now recovered processing of SQS queues via Lambda Event Source Mappings. We are now working through processing the backlog of SQS messages in Lambda queues. [05:48 AM PDT] We are making progress on resolving the issue with new EC2 instance launches in the US-EAST-1 Region and are now able to successfully launch new instances in some Availability Zones. We are applying similar mitigations to the remaining impacted Availability Zones to restore new instance launches. As we continue to make progress, customers will see an increasing number of successful new EC2 launches. We continue to recommend that customers launch new EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. [07:29 AM PDT] We have confirmed multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for the connectivity issues and are continuing to investigate the root cause. [08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations. [09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. [10:03 AM PDT] We continue to apply mitigation steps for network load balancer health and recovering connectivity for most AWS services. Lambda is experiencing function invocation errors because an internal subsystem was impacted by the network load balancer health checks. We are taking steps to recover this internal Lambda system. For EC2 launch instance failures, we are in the process of validating a fix and will deploy to the first AZ as soon as we have confidence we can do so safely. We will provide an update by 10:45 AM PDT. [10:38 AM PDT] Our mitigations to resolve launch failures for new EC2 instances are progressing and the internal subsystems of EC2 are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. We are applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside. [11:22 AM PDT] Our mitigations to resolve launch failures for new EC2 instances continue to progress and we are seeing increased launches of new EC2 instances and decreasing networking connectivity issues in the US-EAST-1 Region. We are also experiencing significant improvements to Lambda invocation errors, especially when creating new execution environments (including for Lambda@Edge invocations). We will provide an update by 12:00 PM PDT. [12:15 PM PDT] We continue to observe recovery across all AWS services, and instance launches are succeeding across multiple Availability Zones in the US-EAST-1 Regions. For Lambda, customers may face intermittent function errors for functions making network requests to other services or systems as we work to address residual network connectivity issues. To recover Lambda’s invocation errors, we slowed down the rate of SQS polling via Lambda Event Source Mappings. We are now increasing the rate of SQS polling as we experience more successful invocations and reduced function errors. We will provide another update by 1:00 PM PDT. [01:03 PM PDT] Service recovery across all AWS services continues to improve. We continue to reduce throttles for new EC2 Instance launches in the US-EAST-1 Region that were put in place to help mitigate impact. Lambda invocation errors have fully recovered and function errors continue to improve. We have scaled up the rate of polling SQS queues via Lambda Event Source Mappings to pre-event levels. We will provide another update by 1:45 PM PDT. [01:52 PM PDT] We have continued to reduce throttles for EC2 instance launches in the US-EAST-1 Region and we continue to make progress toward pre-event levels in all Availability Zones (AZs). AWS services such as ECS and Glue, which rely on EC2 instance launches will recover as the successful instance launch rate improves. We see full recovery for Lambda invocations and are working through the backlog of queued events which we expect to be full processed in approximately in the next two hours. We will provide another update by 2:30 PM PDT. [02:48 PM PDT] We have restored EC2 instance launch throttles to pre-event levels and EC2 launch failures have recovered across all Availability Zones in the US-EAST-1 Regions. AWS services which rely on EC2 instance launches such as Redshift are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours. We can confirm that Connect is handling new voice and chat sessions normally. There is a backlog of analytics and reporting data that we must process and anticipate that we will have worked through the backlog over the next two hours. We will provide an update by 3:30 PM PDT.

2. monitoring Oct 20, 2025, 01:47 PM UTC

   We are continuing to monitor for any further issues.

3. monitoring Oct 20, 2025, 02:33 PM UTC

   We are continuing to monitor for any further issues.

4. monitoring Oct 20, 2025, 03:58 PM UTC

   We are continuing to monitor for any further issues.

5. monitoring Oct 20, 2025, 04:15 PM UTC

   We are continuing to monitor for any further issues.

6. monitoring Oct 20, 2025, 05:05 PM UTC

   We are continuing to monitor for any further issues.

7. monitoring Oct 20, 2025, 05:40 PM UTC

   We are continuing to monitor for any further issues.

8. monitoring Oct 20, 2025, 06:28 PM UTC

   We are continuing to monitor for any further issues.

9. monitoring Oct 20, 2025, 07:19 PM UTC

   We are continuing to monitor for any further issues.

10. monitoring Oct 20, 2025, 08:05 PM UTC

    We are continuing to monitor for any further issues.

11. monitoring Oct 20, 2025, 09:01 PM UTC

    We are continuing to monitor for any further issues.

12. monitoring Oct 20, 2025, 10:39 PM UTC

    We are continuing to monitor for any further issues.

13. resolved Oct 21, 2025, 12:15 AM UTC

    This incident has been resolved.

14. postmortem Oct 21, 2025, 11:20 AM UTC

    \[RESOLVED\] Increased Error Rates and Latencies **Oct 20 3:53 PM PDT** Between 11:49 PM PDT on October 19 and 2:24 AM PDT on October 20, we experienced increased error rates and latencies for AWS Services in the US-EAST-1 Region. Additionally, services or features that rely on US-EAST-1 endpoints such as IAM and DynamoDB Global Tables also experienced issues during this time. At 12:26 AM on October 20, we identified the trigger of the event as DNS resolution issues for the regional DynamoDB service endpoints. After resolving the DynamoDB DNS issue at 2:24 AM, services began recovering but we had a subsequent impairment in the internal subsystem of EC2 that is responsible for launching EC2 instances due to its dependency on DynamoDB. As we continued to work through EC2 instance launch impairments, Network Load Balancer health checks also became impaired, resulting in network connectivity issues in multiple services such as Lambda, DynamoDB, and CloudWatch. We recovered the Network Load Balancer health checks at 9:38 AM. As part of the recovery effort, we temporarily throttled some operations such as EC2 instance launches, processing of SQS queues via Lambda Event Source Mappings, and asynchronous Lambda invocations. Over time we reduced throttling of operations and worked in parallel to resolve network connectivity issues until the services fully recovered. By 3:01 PM, all AWS services returned to normal operations. Some services such as AWS Config, Redshift, and Connect continue to have a backlog of messages that they will finish processing over the next few hours. We will share a detailed AWS post-event summary.

Read the full incident report →

Started May 09, 2025, 12:57 PM UTC· resolved May 09, 2025, 01:32 PM UTC

1. investigating May 09, 2025, 12:57 PM UTC

   integrador de sms en venezuela esta presentando problemas en servidor Le saludamos respetuosamente e informamos que, nuestro Integrador de SMS en Venezuela esta presentando un incidente sobre su Data Center, debido a una falla en uno de sus enlaces principales, generando problema de conexión, situación que impacta en el despacho de los mensajes de notificación (MT) y bidireccional o interactivo (MO) que son emitidos hacia las operadoras locales en Venezuela. Nuestro integrador se encuentra en comunicación con el personal operativo de Data Center. Una vez solventada la falla estaremos informando.

2. monitoring May 09, 2025, 01:20 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 09, 2025, 01:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 02, 2025, 07:14 PM UTC· resolved May 07, 2025, 05:34 PM UTC

1. investigating Apr 02, 2025, 07:14 PM UTC

   Se ha detectado un incidente en uno de nuestros servidores de despacho, el cual está generando retrasos en el envío de correos electrónicos. Estamos trabajando para resolver este problema con el equipo del Centro de Datos lo antes posible y les mantendremos informados sobre cualquier actualización. Agradecemos su comprensión y paciencia

2. identified Apr 07, 2025, 06:02 PM UTC

   Actualización sobre la Migración del Nodo y Restitución del Servicio Nos gustaría informarles sobre el progreso en la migración del nodo hacia nuestro nuevo centro de datos. Actualmente, estamos en una fase crítica del proceso que implica el movimiento de bloques de direcciones IPs. Esta actividad está siendo ejecutada por nuestro proveedor y el Proveedor de Servicios de Internet (ISP). Migración del Nodo: Se está llevando a cabo la migración para mejorar la infraestructura y la estabilidad del servicio. Movimiento de Direcciones IP: Este proceso es esencial para asegurar una transición fluida y evitar interrupciones en el servicio. Nuestro principal objetivo es restituir el servicio lo más pronto posible y minimizar el impacto a nuestros usuarios. Apreciamos su comprensión y paciencia mientras llevamos a cabo este proceso crítico. Les aseguramos que estamos trabajando diligentemente para restablecer todos los servicios a la brevedad. Les informaremos sobre cualquier avance significativo en este asunto.

3. monitoring Apr 15, 2025, 12:21 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 07, 2025, 05:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 05, 2024, 06:06 PM UTC· resolved Nov 05, 2024, 08:16 PM UTC

1. investigating Nov 05, 2024, 06:06 PM UTC

   We are currently investigating this issue.

2. identified Nov 05, 2024, 07:30 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Nov 05, 2024, 08:03 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 05, 2024, 08:16 PM UTC

   This incident has been resolved.

5. postmortem Nov 06, 2024, 03:40 PM UTC

   Descargar informe de reporte en español: [https://drive.google.com/file/d/1IiEQArry8AjBU4M-g6TPHQ95j-iK7EXo/view?usp=sharing](https://drive.google.com/file/d/1IiEQArry8AjBU4M-g6TPHQ95j-iK7EXo/view?usp=sharing) ‌ **Incident Report** **Incident ID:**CYMSLP-126-241105 **Date:**11/05/2024 ### **INCIDENT DESCRIPTION** **Incident reported by:** DevOps **Date and time of the incident:** 11/05/2024 - 12:30 PM \(GMT-4\) **Time elapsed from when the DevOps team noticed the incident until its resolution:** 12:30 PM - 4:00 PM \(GMT-4\) **3 hours 30 minutes** **Details about the services affected by the incident:** * High degradation in communication delivery speed.There was prolonged queuing in the delivery of communications across various channels, including email, SMS, and push notifications. * Both mass campaigns and transactional notifications were affected. * A group of platform users reported that communication processing was halted. **Severity of the incident \(Critical, High, Medium, Low\):**Critical. 3 hours 30 minutes since the initial incident report. **Frequency of this type of incident:**Very low: First occurrence in 15 years. ‌ ### **CAUSE OF THE INCIDENT** **Details about the vulnerability that caused the incident:** A very high number of write accesses to the tables managing the error logs of the communication orchestrator were detected. This occurs when a message cannot be processed for some reason. The most common reasons include: the recipient's address is invalid and the processing route is undefined. Although logging errors is a normal process within the platform, the massive access caused a bottleneck in writing to the table, halting message processing. **Example of contained query:**  **Degradation of insertion times in the error log:**  It was determined that the failure originated from the simultaneous activation of 8 conversations, totaling approximately 8 million messages with destination addresses in an invalid or empty format.  ‌ ‌ ### **CORRECTIVE ACTIONS** 1. Volume of transactions to process within the orchestrator verified by DevOps 12:30 PM \(GMT-4\) 2. Transaction slowness and wait times in the Database verified by DevOps 12:35 PM \(GMT-4\) 3. DANAconnect team was notified via Slack on the Incident channel 12:45 PM \(GMT-4\) 4. The injection of new activations that could be affected was paused 12:50 PM \(GMT-4\) 5. All clients were notified about the platform incident and posted on[ https://status.danaconnect.com](https://status.danaconnect.com) 1:06 PM \(GMT-4\) 6. A Security Snapshot of the Database Cluster was generated 1:10 PM \(GMT-4\) 7. It was determined that the origin of the platform degradation was the write access to the error log 1:56 PM \(GMT-4\) 8. The conversations causing the massive generation of error logs were located 2:51 PM \(GMT-4\) 9. The conversations that initiated the incident to prevent writing to the error log were stopped3:00 PM \(GMT-4\) 10. The corrective action did not improve platform performance to the expected level 3:10 PM \(GMT-4\) 11. It was decided to initiate a general truncation/cleanup of the queue accumulating the processing of messages that caused the incident3:30 PM \(GMT-4\) 12. All API/WS and Orchestrator services were restarted 3:50 PM \(GMT-4\) 13. An internal testing cycle by the DevOps team was initiated 3:55 PM \(GMT-4\) 14. The resolution of the incident was confirmed to the entire DANAconnect team internally via Slack - Incident Channel about 4:00 PM \(GMT-4\) **Details about the solution/patch implemented to resolve the incident:** * Cleaned up the messages that caused the failure at the processing queue level. * Test reports clearly indicate that the implemented solution is functioning. **Database during transaction processing / Incident errors**12:30 PM \(GMT-4\)_Note: The time on the graphs is local in AWS._  ‌ **Database after queue cleanup:**Processing was fully restored4:00 PM \(GMT-4\)_Note: The time on the graphs is local in AWS._  ‌ ‌ ### **ACTIONS TO PREVENT THE INCIDENT FROM RECURRING** **Specialized Monitoring:** * Configuring a special dashboard and alerts for excessive access to the error log table. This will alert the DevOps team to analyze the need for preventive or corrective action. * Monitoring dashboard for conversations writing errors: This will quickly identify which conversations may be generating massive bursts of writes to the error log. **Solution Changes:** * We are considering configuring a parameter that can be adjusted on-demand to disable writing to the error log. **Documentation:** * Added this new type of case and its solution to the Incident Response Plan to provide a quick and effective response.

Read the full incident report →