Datacake incident

MQTT processing delays

Minor Resolved View vendor source →

Datacake experienced a minor incident on April 9, 2025, lasting —. The incident has been resolved; the full update timeline is below.

Started
Apr 09, 2025, 05:24 AM UTC
Resolved
Apr 09, 2025, 05:24 AM UTC
Duration
Detected by Pingoru
Apr 09, 2025, 05:24 AM UTC

Update timeline

  1. resolved Apr 09, 2025, 05:24 AM UTC

    Type: Incident Duration: 6 hours and 42 minutes Affected Components: MQTT Server Apr 9, 05:24:20 GMT+0 - Investigating - Our MQTT broker is experiencing processing delays. No data has been lost, and the system should be caught up shortly. Apr 9, 12:06:42 GMT+0 - Resolved - Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future.