Gemfury incident

Platform issues causing instability

Major Resolved View vendor source →

Gemfury experienced a major incident on August 13, 2021 affecting Dashboard and Badge, lasting 5h 10m. The incident has been resolved; the full update timeline is below.

Started
Aug 13, 2021, 12:14 PM UTC
Resolved
Aug 13, 2021, 05:24 PM UTC
Duration
5h 10m
Detected by Pingoru
Aug 13, 2021, 12:14 PM UTC

Affected components

DashboardBadge

Update timeline

  1. investigating Aug 13, 2021, 12:14 PM UTC

    We are investigating consistent internal errors for our Content Explorer backend.

  2. investigating Aug 13, 2021, 12:17 PM UTC

    The problem also seems to extend to Dashboard live updates, and Version Badge public service.

  3. identified Aug 13, 2021, 03:17 PM UTC

    The issue was identified as a partial outage for our platform provider. We cannot address this issue directly, but we were able to implement a bypass and partially return service to Version Badge and Content Explorer. It may take clients an hour or two to see the Version Badge return due to DNS caching. We are still working to restore Dashboard live updates.

  4. monitoring Aug 13, 2021, 03:24 PM UTC

    Dashboard live updates have been restored as well, subject to the same DNS caching delays. We're going to continue to monitor error rates as the clients transition to the new endpoint.

  5. resolved Aug 13, 2021, 05:24 PM UTC

    Confirmed as resolved.