Gemfury had 12 outages in the last 2 years totaling 47h 10m of downtime — averaging 0.5 incidents per month.
There were 12 Gemfury outages since August 17, 2024 totaling 47h 10m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating this issue.
Bots/crawlers are overwhelming the service. Adjusting throttling.
Throttling adjustments are in place. Errors subsiding. Monitoring.
Errors have returned to normal levels. Fix confirmed. Resolving.
We're seeing unusual traffic to our repositories that are causing elevated errors and timeouts. We've investigating possible ways to miteigate this.
The number of open connections to each server was reaching our configured limit. We have raised the limit. Monitoring.
Configuration & scaling improvements seem to be absorbing the traffic spikes more comfortably now. Resolving.
We are again seeing errors & delays for uploads. Investigating.
Looks like some of the misconfiguration issues persisted from yesterday. A few of the queues were not processed, which prevented account updates from propagating to repositories. We have made a fix, and are now processing the backlog.
Jobs processing normally, and outstanding work has been completed. Resolving.
We are investigating delays in processing of uploads and other asynchronous work.
Production-specific misconfiguration caused our async workers to enter a crash-restart loop. We've fixed the issue, and we will be looking into how to catch these issues earlier in the release process. Resolving.
We are currently investigating this issue.
We had a temporary 10x traffic increase on one of our backend services, which lead to elevated timeouts and errors. Error rates have now returned to normal while we continue to look for the root cause of that traffic spike.
Resolving the incident to continue the investigation and fix offline.
We are investigating elevated error and timeout rates.
Unusual traffic spike seems to have temporarily overwhelmed one of our API endpoints. That spike has subsided, error/timeout rates are back to normal. We are monitoring while we continue to investigate.
We've tracked the issue down to a frequent DB update and applied a fix to reduce it. Resolving the incident while we will keep monitoring the fix.
We are seeing elevated error rates. Investigating.
We've found that the issue was triggered by a recent configuration update. We've rolled back the changes, and the error rates have returned to normal levels. Checking further into the changes that caused the failure.
It appears a misplaced space in a configuration update of caused the configured library to throw exceptions for some of the requests. We've fixed the configuration and will be patching the library to ensure this is handled more gracefully. Resolving.
We are seeing increased timeouts for proxied requests to the public npm registry. We are tracking the upstream issue with npmjs.org.
Upstream incident has been resolved, and no problems otherwise. Resolving.
We are currently investigating this issue.
Our asynchronous workers are not running due to platform issues, and thus cannot process uploads. We are waiting for updates from our platform provider.
We were able to start some of the workers. Starting to work through the async task backlog.
Although some platform instability persists, we have processed most of the asynchronous task backlog (uploads, emails, etc.). Most functionality has been restored. We will continue monitoring the status of the platform until resolution. If you're still experiencing delays or issues with your account, please contact [email protected] with full details.
We are continuing to monitor for any further issues.
We are resolving this as we are no longer seeing customer-facing errors. We'll continue to monitor both our and the platform's status to address any issues that may arise.
We've deployed an update that caused elevated errors due to a caching error that didn't surface during testing. We've rolled back the update, implemented a fix, and rolled out the fixed update. Monitoring.
The fix was effective. Service fully restored.
Investigating custom domains and uploads outage.
A routine infrastructure update misconfigured our load balancers. We've updated the load balancers and updated DNS settings. It may take time for changes to propagate to clients. Monitoring.
Request and error rates returning to normal levels. We will continue to monitor. Resolving.