Datacake incident

MQTT processing delays

Resolved after 6 hours 42 minutes, on April 9, 2025.

resolved Apr 09, 2025, 12:06 PM UTC

Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future..

Started
Apr 09, 2025, 05:24 AM UTC
Resolved
Apr 09, 2025, 12:06 PM UTC
Duration
6h 42m
Detected by Pingoru
Apr 09, 2025, 05:24 AM UTC

Update timeline

  1. investigating Apr 09, 2025, 05:24 AM UTC

    Our MQTT broker is experiencing processing delays. No data has been lost, and the system should be caught up shortly..

  2. resolved Apr 09, 2025, 12:06 PM UTC

    Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future..

Your product runs on more than Datacake

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →