Ngrok incident

OAuth, OIDC, and SAML modules for Edges degraded

Major Resolved View vendor source →

Ngrok experienced a major incident on February 7, 2024 affecting Region - AP and Region - AU and 1 more component, lasting 25m. The incident has been resolved; the full update timeline is below.

Started
Feb 07, 2024, 07:43 PM UTC
Resolved
Feb 07, 2024, 08:09 PM UTC
Duration
25m
Detected by Pingoru
Feb 07, 2024, 07:43 PM UTC

Affected components

Region - APRegion - AURegion - EURegion - INRegion - JPRegion - SARegion - USRegion - us-cal-1

Update timeline

  1. identified Feb 07, 2024, 07:43 PM UTC

    We have identified an issue with the following Edge modules that may be impacting customers' ability to access content served by the Edge: OAuth, OIDC, and SAML. We are rolling out a fix and monitoring. This issue started approximately ten minutes ago.

  2. monitoring Feb 07, 2024, 07:59 PM UTC

    Fixes are rolled out. The team will continue monitoring to confirm a full recovery.

  3. monitoring Feb 07, 2024, 08:04 PM UTC

    Customers using SAML modules may in rare cases need to clear cookies if you are seeing `BAD REQUEST` errors on login.

  4. resolved Feb 07, 2024, 08:09 PM UTC

    The system should be recovered for all users.