Runpanther incident

September 20, 2022 GitHub API Gateway Errors

Major Resolved View vendor source →

Runpanther experienced a major incident on September 20, 2022 affecting Data Ingestion into Panther (Log Processing), lasting 8h 3m. The incident has been resolved; the full update timeline is below.

Started
Sep 20, 2022, 06:52 AM UTC
Resolved
Sep 20, 2022, 02:55 PM UTC
Duration
8h 3m
Detected by Pingoru
Sep 20, 2022, 06:52 AM UTC

Affected components

Data Ingestion into Panther (Log Processing)

Update timeline

  1. investigating Sep 20, 2022, 06:52 AM UTC

    Panther has identified an issue with the GitHub Audit Log puller that started at 2:03 am UTC. The GitHub endpoint is returning a 502 Gateway Error for all requests. This is impacting data ingestion for all customers using the GitHub Audit Log Puller. Panther’s team is currently investigating the root cause of this problem with GitHub and we will update this incident as our investigation and remediation process progresses.

  2. identified Sep 20, 2022, 12:33 PM UTC

    GitHub has opened an incident (https://www.githubstatus.com/incidents/v5bcd4pzckky) reporting that their audit log REST API requests are currently disabled and they are working on a fix. This is the root cause of the Panther GitHub log puller errors. We will update this incident when more information is available.

  3. resolved Sep 20, 2022, 02:55 PM UTC

    GitHub has updated their incident (https://www.githubstatus.com/incidents/v5bcd4pzckky) to indicate that the audit log API is now functioning as intended. Our monitoring indicates that all customer accounts with GitHub log pullers are fully operational. Data that was delayed as a result of this incident has already been successfully backfilled for all customers.