EMQ Technologies incident

Partial Node Failure in US Region Serverless Cluster

Minor Resolved View vendor source →

EMQ Technologies experienced a minor incident on July 17, 2025 affecting EMQX Serverless, lasting 53m. The incident has been resolved; the full update timeline is below.

Started
Jul 17, 2025, 02:02 AM UTC
Resolved
Jul 17, 2025, 02:56 AM UTC
Duration
53m
Detected by Pingoru
Jul 17, 2025, 02:02 AM UTC

Affected components

EMQX Serverless

Update timeline

  1. investigating Jul 17, 2025, 02:02 AM UTC

    At 01:56 UTC on July 17, 2025, we detected a partial node failure in the Serverless cluster in the US region due to underlying instance issues. We immediately initiated a failover process to replace the affected nodes. During the failover, some clients connected to the impacted nodes may experience brief disconnections. However, client reconnections are not affected, and services are expected to recover automatically. We are closely monitoring the situation and will provide further updates as needed.

  2. identified Jul 17, 2025, 02:03 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Jul 17, 2025, 02:09 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Jul 17, 2025, 02:49 AM UTC

    We are continuing to monitor for any further issues.

  5. resolved Jul 17, 2025, 02:56 AM UTC

    By 02:09 UTC, the failover was successfully completed. Based on continued monitoring, we confirm that the Serverless cluster has fully recovered and is operating normally. We appreciate your patience and understanding.