Performance degradation in Cloud Chat instance 1

Update timeline

1. investigating Sep 01, 2025, 01:30 PM UTC

   We are currently investigating this issue and will provide updates here as they become available

2. investigating Sep 01, 2025, 01:48 PM UTC

   We identified that the overloaded resource is one of our workers (Sidekiq), which received a sudden surge in messages and is therefore processing tasks with delays. Two teams are already working in parallel: one provisioning additional infrastructure and the other optimizing this worker’s logic.

3. investigating Sep 01, 2025, 02:24 PM UTC

   We are about to roll out a new, more optimized message intake logic. The deployment is expected to take place within the next 30 minutes.

4. investigating Sep 01, 2025, 03:34 PM UTC

   We are continuing to optimize message intake performance while gradually scaling up the infrastructure. At the moment, the average processing time for incoming messages on Cloud Chat instance-1 is around 50 minutes.

5. investigating Sep 01, 2025, 04:27 PM UTC

   With the recent infrastructure adjustments and performance improvements, events are now being processed faster than they arrive. As a result, the application should normalize soon, and new messages will be ingested increasingly closer to real time.

6. investigating Sep 01, 2025, 05:18 PM UTC

   Impact: A sudden ~4x increase in message volume caused significant delays in processing external events (WhatsApp webhooks, email, etc.), impacting the application’s ingestion time. Root Cause: The bottleneck occurred in a shared infrastructure resource that was not scaled to handle the sudden traffic spike. In addition, the notifications feature contributed to excessive resource consumption. Resolution: Infrastructure upgrades and optimizations to the event intake logic were implemented, allowing the system to resume real-time message processing. As a temporary measure, the notifications feature was disabled. Processing times are now normalized, but the team continues to closely monitor the infrastructure and will keep working on further performance improvements.

7. monitoring Sep 01, 2025, 06:46 PM UTC

   Processing times are now normalized, but the team continues to closely monitor the infrastructure and will keep working on further performance improvements.

8. resolved Sep 01, 2025, 07:52 PM UTC

   ✅ After the infrastructure upgrades and applied improvements, message processing has returned to normal and is now happening in real time. We will continue to closely monitor performance and apply additional optimizations to ensure ongoing stability.