Pusher incident

We are experiencing a major outage in our US2 cluster

Critical Resolved View vendor source →

Pusher experienced a critical incident on September 8, 2024 affecting Channels REST API and Channels WebSocket client API and 1 more component, lasting 5h 5m. The incident has been resolved; the full update timeline is below.

Started
Sep 08, 2024, 06:00 PM UTC
Resolved
Sep 08, 2024, 11:06 PM UTC
Duration
5h 5m
Detected by Pingoru
Sep 08, 2024, 06:00 PM UTC

Affected components

Channels REST APIChannels WebSocket client APIChannels presence channelsChannels Webhooks

Update timeline

  1. investigating Sep 08, 2024, 06:00 PM UTC

    We are investigating what is causing the issue to our US cluster.

  2. investigating Sep 08, 2024, 07:31 PM UTC

    We are escalating the current issue from a partial outage to a major outage. The majority of traffic in the US2 cluster is currently affected. Our team is fully engaged and working to resolve the issue.

  3. investigating Sep 08, 2024, 09:19 PM UTC

    Our team is still investigating the issue within our infrastructure that is preventing us from adding WebSocket capacity to the US2 cluster. This issue is specifically impacting our WebSocket deployments. The majority of clients are currently unable to connect using WebSocket. However fallback mechanisms (SockJS, HTTP streaming, and HTTP polling) are available for connection if they are not disabled in your configuration.

  4. monitoring Sep 08, 2024, 10:19 PM UTC

    The team has been able to identify the issue and push a fix. We are seeing a full recovery of the services. We will continue to monitor.

  5. monitoring Sep 08, 2024, 10:21 PM UTC

    A fix has been implemented, and we are monitoring the results. WebSocket connections should now be functional.

  6. monitoring Sep 08, 2024, 11:05 PM UTC

    We are continuing to monitor for any further issues.

  7. resolved Sep 08, 2024, 11:06 PM UTC

    This incident has been resolved.