Gemfury incident

Networking errors in upload hosts

Major Resolved View vendor source →

Gemfury experienced a major incident on July 17, 2020 affecting Repo endpoints and Uploads, lasting 2h 58m. The incident has been resolved; the full update timeline is below.

Started
Jul 17, 2020, 09:21 PM UTC
Resolved
Jul 18, 2020, 12:20 AM UTC
Duration
2h 58m
Detected by Pingoru
Jul 17, 2020, 09:21 PM UTC

Affected components

Repo endpointsUploads

Update timeline

  1. investigating Jul 17, 2020, 09:21 PM UTC

    Investigating elevated networking errors on upload processing hosts

  2. investigating Jul 17, 2020, 09:31 PM UTC

    Outage appears to be caused by sporadic DNS failures to access Gemfury API, and other services. Also note that Cloudflare DNS is currently down, may be related.

  3. investigating Jul 17, 2020, 09:37 PM UTC

    Error rates improving, service returning to normal

  4. investigating Jul 17, 2020, 09:55 PM UTC

    Errors have dissipated. Services running normally. Monitoring for any changes.

  5. resolved Jul 18, 2020, 12:20 AM UTC

    DNS issues have been resolved upstream. Everything is back to normal here. We will be looking into how to mitigate this better in the future.