Python Package Index had 11 outages in the last 2 years totaling 81h 16m of downtime — averaging 0.5 incidents per month.
There were 11 Python Package Index outages since December 13, 2024 totaling 81h 16m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating this issue.
The traffic has subsided and the cluster is stable. A performance fix has been drafted.
A fix has been implemented and the incident is closed.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating why PyPI uploads are failing.
We identified a database lock causing contention and stopped all upload backends to release all locks and are monitoring for recovery.
This incident has been resolved.
We are currently observing a slowdown in background requests processing for events on PyPI.org This could lead to stale web pages and API responses as the events system is not able to purge the caches. Our team is continuing to investigate.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The PyPI search index has degraded and is not returning results. A reindexing job has begun, and results will be available once it completes.
This incident has been resolved.
We are working to investigate a partial outage in the PyPI backends.
We have identified a large increase in CDN cache misses causing excess load on our backends. We are working to determine the reason for this increase.
We have identified a flood of requests from a cluster using the `python-requests/2.32.3` User-Agent, and attempting to hit all JSON project/releasese en masse. This causes excessive cache misses, which are overwhelming our backends. A temporary block has been put in place.
Our block is in place and systems are stable, but we are not satisfied with having to block such a generic user-agent. If you are aware of a system that began a mass crawl of PyPI's JSON release endpoints starting at around 22:05 UTC 2025-05-25, please contact [email protected].
We have removed the temporary block and are continuing to monitor.
We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.
We're investigating elevated error rates for pypi.org
We've identified an issue with database contention arising from locking.
The locking process has been terminated and we're monitoring as things recover.
This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.
We are working to resolve an outage impacting our /simple and /json apis.
We've increased capacity in the worker pools that service /simple and /json and are monitoring as things recover.
Errors have subsided, and we are monitoring as things continue to recover.
This incident has been resolved.
We are investigating issues with files from files.pythonhosted.org being served with corrupted data. A support ticket has been opened with our CDN provider and we are awaiting their response.
This issue appears to be isolated to a single edge node in our CDN provider's network. They have validated the issue and are working to resolve.
Our CDN provider has reported that the node in question has been removed from service. We are monitoring for continued reports.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating an infrastructure outage impacting python.org/downloads backends, docs.python.org backends, bugs.python.org backends, wiki.python.org backends, as well as other services.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Private networking in our datacenter for these services is currently intermittent, we have opened a ticket with the provider requesting assistance.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We have begun seeing recovery across all services as of 14:44 UTC, we are monitoring for stability and ensuring all services are healthy.
This incident has been resolved.