Cloud Humans Outage History

Started Oct 02, 2025, 02:40 PM UTC· resolved Oct 02, 2025, 04:30 PM UTC

1. investigating Oct 02, 2025, 02:40 PM UTC

   🇧🇷 Detectamos uma instabilidade em nossos servidores que pode causar lentidão em alguns acessos ao Hub, Eddie e Cloudchat. Nossa equipe já está investigando a situação. 🇺🇸 We have detected an instability in our servers that may cause slowness in some accesses to Hub, Eddie, and Cloudchat. Our team is already investigating the situation. 🇪🇸 Detectamos una inestabilidad en nuestros servidores que puede causar lentitud en algunos accesos a Hub, Eddie y Cloudchat. Nuestro equipo ya está investigando la situación.

2. identified Oct 02, 2025, 03:54 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Oct 02, 2025, 04:30 PM UTC

   We have identified that the root cause of the issue affecting our systems was related to Cloudflare’s incident involving SSL certificate provisioning delays. 09:26 UTC – Cloudflare began investigating delays in provisioning some SSL certificates. Certificates already in production were unaffected. 11:28 UTC – Cloudflare identified the issue and started implementing a fix. 15:05 UTC – Cloudflare confirmed ongoing work to fully resolve the incident. Although Cloudflare has not yet marked the incident as resolved, the situation has normalized on our end and all systems are currently operational. We continue to monitor closely.

Read the full incident report →

Started Sep 11, 2025, 09:48 PM UTC· resolved Sep 11, 2025, 11:35 PM UTC

1. identified Sep 11, 2025, 09:48 PM UTC

   🇧🇷 Português [Interrupção Parcial] Cloud Chat Início: 16:30 EST / 17:30 BRT Estamos enfrentando uma indisponibilidade parcial no Cloud Chat que afeta alguns usuários de clientes. Os problemas identificados são: Alguns usuários não conseguem abrir o Cloud Chat. Alguns clientes não estão recebendo mensagens no Cloud Chat (as mensagens foram enviadas para o cliente, mas não estão sendo entregues na plataforma). Elas estão entrando no nosso backend e estão em fila esperando para serem processadas. Nossa equipe já mapeou a causa do problema e está trabalhando ativamente em um ajuste. Nas próximas atualizações, publicaremos detalhes sobre a origem da falha e as medidas corretivas. Agradecemos pela compreensão. 🇺🇸 English [Partial Outage] Cloud Chat Start: 4:30 PM EST / 5:30 PM BRT We are currently experiencing a partial outage in Cloud Chat affecting some client users. The identified issues are: Some users are unable to open Cloud Chat. Some clients are not receiving messages in Cloud Chat (messages have been sent to the client but are not being delivered on the platform). They are entering our backend and are queued, waiting to be processed. Our team has already identified the root cause and is actively working on a fix. In the next updates, we will share details about the issue and the corrective measures. Thank you for your understanding.

2. identified Sep 11, 2025, 10:02 PM UTC

   🇧🇷 Português Atualização sobre o incidente no Cloud Chat Identificamos que a instabilidade ocorreu após a liberação de uma nova funcionalidade integrada ao WhatsApp. Essa atualização impactou parte do sistema de controles, afetando tanto o processamento de mensagens quanto o acesso de alguns usuários. Assim que percebemos o problema, realizamos um rollback para a versão estável e o consumo das mensagens em fila foi finalizado às 18:43 BRT / 17:43 EST. Neste momento, estamos normalizando gradualmente o acesso dos usuários ao Cloud Chat. O processo de restauração completa acontece de forma progressiva, com estimativa de normalização total em até 15 minutos. Seguiremos monitorando de perto e traremos novas atualizações aqui. Agradecemos pela paciência e compreensão. ⸻ 🇺🇸 English Update on the Cloud Chat Incident We identified that the instability occurred after the release of a new WhatsApp-related feature. This update impacted part of our control systems, affecting both message processing and user access for some clients. Once detected, we rolled back to the stable version, and the processing of queued messages was completed at 5:43 PM EST / 6:43 PM BRT. We are now gradually restoring user access to Cloud Chat. The full restoration process is progressive, with an estimated completion time of up to 15 minutes. We will continue to closely monitor the situation and provide updates here. Thank you for your patience and understanding.

3. identified Sep 11, 2025, 10:12 PM UTC

   🇧🇷 Português Atualização sobre o incidente no Cloud Chat Metade dos pods afetados já foi normalizada, o que significa que pelo menos metade dos usuários impactados já deve ter recuperado o acesso. A previsão é que a normalização completa aconteça em aproximadamente 5 minutos, à medida que o processo é concluído para todos os pods. Seguiremos monitorando de perto e atualizaremos aqui assim que o incidente for totalmente resolvido. ⸻ 🇺🇸 English Update on the Cloud Chat Incident Half of the affected pods have already been restored, which means that at least half of the impacted users should have regained access. We estimate that full normalization will be completed within the next 5 minutes, as the process finishes across all pods. We will continue closely monitoring and will update here once the incident is fully resolved.

4. identified Sep 11, 2025, 10:19 PM UTC

   🇧🇷 Português Atualização sobre o incidente no Cloud Chat Mais de 90% dos acessos já foram restabelecidos. Desde o reprocessamento total das mensagens em fila, concluído às 18:43 BRT / 17:43 EST, a ClaudIA já estava respondendo normalmente. O que permanecia pendente era apenas o acesso ao front-end de alguns usuários, que está sendo gradualmente normalizado. Seguiremos acompanhando até a completa resolução. ⸻ 🇺🇸 English Update on the Cloud Chat Incident Over 90% of user access has already been restored. Since the full reprocessing of queued messages was completed at 5:43 PM EST / 6:43 PM BRT, ClaudIA has been responding normally. The only remaining effect was user front-end access, which is now being gradually normalized. We will continue monitoring until full resolution.

5. monitoring Sep 11, 2025, 11:04 PM UTC

   🇧🇷 Português [Resolvido] Incidente no Cloud Chat O incidente foi totalmente normalizado às 19:23 BRT / 18:23 EST, e todos os usuários já têm acesso restabelecido. Nossa equipe segue monitorando para garantir estabilidade. Resumo do ocorrido: • Uma atualização relacionada ao WhatsApp, liberada às 17:30 BRT, impactou o processamento de mensagens em alguns canais de clientes e afetou o acesso de parte dos usuários. • Assim que identificado, realizamos rollback para a versão estável, o que normalizou o processamento das mensagens às 18:43 BRT. • Para restabelecer totalmente o acesso dos usuários, foi necessário um ajuste adicional após o rollback, o que exigiu uma atualização progressiva em todos os pods. Esse processo foi concluído às 19:23 BRT. Próximos passos: Estamos implementando melhorias de arquitetura (Argo CD e Blue-Green Deploy no Cloud Chat, já utilizados em outras aplicações) que aumentarão a resiliência e evitarão incidentes semelhantes. Agradecemos a paciência e compreensão. ⸻ 🇺🇸 English [Resolved] Cloud Chat Incident The incident was fully resolved at 6:23 PM EST / 7:23 PM BRT, and all users now have access restored. Our team continues to monitor to ensure stability. Summary of what happened: • A WhatsApp-related update released at 5:30 PM BRT impacted message processing in some client channels and temporarily affected user access. • Once detected, we rolled back to the stable version, which restored message processing by 5:43 PM EST. • To fully restore user access, an additional adjustment was required after the rollback, involving a progressive rollout across all pods. This was completed by 6:23 PM EST. Next steps: We are implementing infrastructure improvements (Argo CD and Blue-Green Deploy for Cloud Chat, already used in other applications) to increase resilience and prevent similar incidents. Thank you for your patience and understanding.

6. resolved Sep 11, 2025, 11:35 PM UTC

   🇧🇷 Português [Resolvido] Incidente no Cloud Chat O incidente foi resolvido às 19:23 BRT / 18:23 EST. Após mais de uma hora de monitoramento, não identificamos novos problemas nos logs nem recebemos relatos de clientes. ⸻ 🇺🇸 English [Resolved] Cloud Chat Incident The incident was resolved at 6:23 PM EST / 7:23 PM BRT. After more than an hour of monitoring, no further issues were detected in the logs and no problems were reported by clients.

Read the full incident report →

Started Sep 03, 2025, 03:25 PM UTC· resolved Sep 03, 2025, 04:25 PM UTC

1. identified Sep 03, 2025, 03:25 PM UTC

   We are currently experiencing some instabilities in our infrastructure after adjustments made in AWS to improve performance. We are already in direct contact with AWS to normalize the situation. This issue is affecting access to Hub and Cloud Chat, and may also cause some slowness in Claudia. Our forecast is that stability will be restored soon, and we will keep you updated.

2. monitoring Sep 03, 2025, 03:36 PM UTC

   All systems are online and operational. We will continue to monitor closely.

3. resolved Sep 03, 2025, 04:25 PM UTC

   The instability we experienced has already been resolved! It occurred during an infrastructure improvement carried out together with our AWS expert consultants.

Read the full incident report →

Started Sep 01, 2025, 01:30 PM UTC· resolved Sep 01, 2025, 07:52 PM UTC

1. investigating Sep 01, 2025, 01:30 PM UTC

   We are currently investigating this issue and will provide updates here as they become available

2. investigating Sep 01, 2025, 01:48 PM UTC

   We identified that the overloaded resource is one of our workers (Sidekiq), which received a sudden surge in messages and is therefore processing tasks with delays. Two teams are already working in parallel: one provisioning additional infrastructure and the other optimizing this worker’s logic.

3. investigating Sep 01, 2025, 02:24 PM UTC

   We are about to roll out a new, more optimized message intake logic. The deployment is expected to take place within the next 30 minutes.

4. investigating Sep 01, 2025, 03:34 PM UTC

   We are continuing to optimize message intake performance while gradually scaling up the infrastructure. At the moment, the average processing time for incoming messages on Cloud Chat instance-1 is around 50 minutes.

5. investigating Sep 01, 2025, 04:27 PM UTC

   With the recent infrastructure adjustments and performance improvements, events are now being processed faster than they arrive. As a result, the application should normalize soon, and new messages will be ingested increasingly closer to real time.

6. investigating Sep 01, 2025, 05:18 PM UTC

   Impact: A sudden ~4x increase in message volume caused significant delays in processing external events (WhatsApp webhooks, email, etc.), impacting the application’s ingestion time. Root Cause: The bottleneck occurred in a shared infrastructure resource that was not scaled to handle the sudden traffic spike. In addition, the notifications feature contributed to excessive resource consumption. Resolution: Infrastructure upgrades and optimizations to the event intake logic were implemented, allowing the system to resume real-time message processing. As a temporary measure, the notifications feature was disabled. Processing times are now normalized, but the team continues to closely monitor the infrastructure and will keep working on further performance improvements.

7. monitoring Sep 01, 2025, 06:46 PM UTC

   Processing times are now normalized, but the team continues to closely monitor the infrastructure and will keep working on further performance improvements.

8. resolved Sep 01, 2025, 07:52 PM UTC

   ✅ After the infrastructure upgrades and applied improvements, message processing has returned to normal and is now happening in real time. We will continue to closely monitor performance and apply additional optimizations to ensure ongoing stability.

Read the full incident report →

Started Aug 23, 2025, 12:00 AM UTC· resolved Aug 23, 2025, 12:00 AM UTC

1. resolved Aug 23, 2025, 12:20 AM UTC

   Impact: During this period, ClaudIA correctly received transcripts of audio and images but did not consider them when generating responses. This led to unexpected behaviors, such as messages like “I’ll listen to the audio and reply,” without actually using the transcription content in the context. Root cause: The issue was introduced in an update published at 2:13 PM today. Resolution: The fix was applied and validated at 8:55 PM. Audio and image processing has returned to normal operation, with ClaudIA now correctly using transcripts as part of the response context.

Read the full incident report →