crates.io incident

Request timeouts/Slow responses

Major Resolved View vendor source →

crates.io experienced a major incident on December 22, 2021 affecting crates.io, lasting 2h 40m. The incident has been resolved; the full update timeline is below.

Started
Dec 22, 2021, 12:27 PM UTC
Resolved
Dec 22, 2021, 03:07 PM UTC
Duration
2h 40m
Detected by Pingoru
Dec 22, 2021, 12:27 PM UTC

Affected components

crates.io

Update timeline

  1. investigating Dec 22, 2021, 12:27 PM UTC

    We are currently investigating an incident that causes slow responses and request timeouts on crates.io.

  2. investigating Dec 22, 2021, 12:37 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Dec 22, 2021, 01:02 PM UTC

    An outage of one of our service providers impacts our primary database and one of our application dynos. Currently, write operations will fail (uploading new crates, ...) and downloads will not be counted. About 25% of all download requests hit the affected dyno and will fail with a timeout. Cargo itself should be mostly unaffected due to retries.

  4. monitoring Dec 22, 2021, 02:39 PM UTC

    Our service provider is still working to restore full functionality, but unfortunately there is no ETA for full recovery at this time. We continue to monitor the situation.

  5. resolved Dec 22, 2021, 03:07 PM UTC

    All of our affected infrastructure has recovered and full functionality has been restored. Some caches may require a bit of warming up, so some slow responses should still be expected.