Gemfury experienced a minor incident on February 12, 2020 affecting Repo endpoints, lasting 3h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Repo endpoints
Update timeline
- investigating Feb 12, 2020, 03:55 PM UTC
Investigating reports of sporadic 403 errors for valid tokens, and possible slowness.
- investigating Feb 12, 2020, 04:03 PM UTC
Seeing spikes up to 9% of requests, from a baseline of 0.5-1%. Rolling back to an earlier build, and disabling experimental optimizations. Bigger accounts may experience greater latencies.
- monitoring Feb 12, 2020, 04:43 PM UTC
403 errors are back below 0.5%, but as predicted, latencies are up. We're going to slowly bring back some optimizations.
- monitoring Feb 12, 2020, 07:24 PM UTC
We were still seeing errors, so we're turning off the optimizations again.
- resolved Feb 12, 2020, 07:41 PM UTC
Everything looks better now. We're going to close the incident while we continue to investigate the root cause.