Ngrok incident

Agent connection failures in US

Major Resolved View vendor source →

Ngrok experienced a major incident on January 11, 2024 affecting Dashboard (https://dashboard.ngrok.com) and Region - US and 1 more component, lasting 3h 14m. The incident has been resolved; the full update timeline is below.

Started
Jan 11, 2024, 12:24 PM UTC
Resolved
Jan 11, 2024, 03:39 PM UTC
Duration
3h 14m
Detected by Pingoru
Jan 11, 2024, 12:24 PM UTC

Affected components

Dashboard (https://dashboard.ngrok.com)Region - USRegion - us-cal-1

Update timeline

  1. investigating Jan 11, 2024, 12:24 PM UTC

    We're investigating an increased failure rate for agents starting tunnels in the US.

  2. investigating Jan 11, 2024, 02:45 PM UTC

    We are continuing to investigate this issue.

  3. investigating Jan 11, 2024, 03:04 PM UTC

    We are rolling out mitigations for the issue and seeing some recovery. Customers may also see errors on dashboard.ngrok.com when attempting to stop or view agent sessions.

  4. monitoring Jan 11, 2024, 03:08 PM UTC

    We believe the issue has been identified and fixed. Customers should see a recovery in the dashboard and our US regions. We're continuing to monitor the situation.

  5. resolved Jan 11, 2024, 03:39 PM UTC

    We are no longer seeing customer impact for this incident.