Python Package Index incident

PyPI Infrastructure Migration

Notice Resolved View vendor source →

Python Package Index experienced a notice incident on January 20, 2014, lasting 5d. The incident has been resolved; the full update timeline is below.

Started
Jan 20, 2014, 10:05 PM UTC
Resolved
Jan 25, 2014, 10:43 PM UTC
Duration
5d
Detected by Pingoru
Jan 20, 2014, 10:05 PM UTC

Update timeline

  1. scheduled Jan 20, 2014, 10:05 PM UTC

    PyPI will be migrated to upgraded infrastructure over the course of approximately 4 hours on 2014-01-25. Package installs via `pip` and downloads will still function during the maintenance. During the migration all uploaded packages will remain available via the `/simple` and `/packages` endpoints. The UI and XMLRPC, JSON, and SSH interfaces will be unavailable. No package registration, upload, or search functionality will be available.

  2. scheduled Jan 25, 2014, 01:00 PM UTC

    Maintenance will begin as scheduled in 60 minutes.

  3. in progress Jan 25, 2014, 02:00 PM UTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  4. in progress Jan 25, 2014, 02:07 PM UTC

    PyPI has been stopped on the existing backend. Failback to our static mirror appears to have been successful. We are beginning to migrate the last of the files and databases to the new infrastructure.

  5. in progress Jan 25, 2014, 03:31 PM UTC

    We're running sanity checks. Testing restored files and databases in the new infrastructure.

  6. monitoring Jan 25, 2014, 04:32 PM UTC

    We've brought PyPI back online on the new Infrastructure! We'll be verifying over the next few hours.

  7. monitoring Jan 25, 2014, 04:50 PM UTC

    It appears search for PyPI is unstable at the moment. We're investigating.

  8. monitoring Jan 25, 2014, 05:20 PM UTC

    Intermittent 500's and Timeouts on the new infrastructure, Working to shake out some bugs.

  9. in progress Jan 25, 2014, 06:12 PM UTC

    We've hit a snag with performance in our new database backend, PyPI will go back to maintenance mode while we upgrade.

  10. in progress Jan 25, 2014, 07:20 PM UTC

    Database upgrade is complete. We're now working on rerouting through our pooling infrastructure for PyPI database calls.

  11. in progress Jan 25, 2014, 07:43 PM UTC

    We have a few small issues to handle. The most obvious being a redirect issue with search. Search results which return one package are intended to redirect to that package, currently this is broken.

  12. monitoring Jan 25, 2014, 09:05 PM UTC

    We're back in the verification stage for the new PyPI infrastructure. Almost complete.

  13. completed Jan 25, 2014, 10:43 PM UTC

    We've completed the migration of PyPI infrastructure!