packagecloud experienced a major incident on October 24, 2025 affecting API and Frontend and 1 more component, lasting 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 24, 2025, 04:12 AM UTC
We are currently investigating an increase in 403 (Forbidden) errors affecting some Packagecloud repositories. Users may experience access denied errors when attempting to read packages. Our team is actively investigating the root cause and will provide updates as more information becomes available.
- investigating Oct 24, 2025, 04:24 AM UTC
We have confirmed it does not affect PyPI and are continuing to investigate the issue.
- investigating Oct 24, 2025, 04:38 AM UTC
This incident has been resolved - we've reverted the change that was causing the issue.
- resolved Oct 24, 2025, 04:40 AM UTC
This incident has been resolved.
- postmortem Nov 07, 2025, 03:08 AM UTC
## Service Impact On October 24th, 2025 from approximately 01:53AM UTC until 04:24AM UTC our package download service experienced an outage affecting customer access. This was confirmed to have impacted Debian and RubyGems. ## Incident Summary At 01:13AM UTC we updated the S3 Bucket Policy for an S3 bucket containing customer packages via Terraform as part of a larger re-architecting project. We received customer reports about Debian package download issues from 01:53AM UTC and started doing isolated, per-customer investigations. At 03:40AM UTC we concluded that the issues were not isolated and declared the incident. More members from our team were called to help investigate the problem, where we found that service for pulling RubyGems had also been impacted. Around 04:00 UTC, the new bucket policy deployed at 01:13AM UTC was identified as a main factor to the access failure and was removed. At 04:19AM UTC our team identified that updating the IAM policy via Terraform did not have the expected result of merging with the pre-existing policy, but rather inadvertently overwrote the pre-existing access policies, causing the loss of access permissions necessary for package downloads. Afterwards it became apparent to our team making these changes that this particular behavior is a [known issue](https://github.com/hashicorp/terraform-provider-aws/issues/6334) with the Terraform AWS provider, which contributed to the difficulty of detection and prevention, thereby increasing the likelihood of the incident occurring for those unaware. After redeploying the pre-existing S3 bucket IAM policy, we confirmed service functionality for Debian package downloads was restored at 04:24 AM UTC. The team subsequently verified that pulling RubyGems and other supported package types were also functioning as expected, and the incident was closed at 04:47 AM UTC. ## Changes we're making We are revising our monitoring approaches to identify and address service disruptions proactively, while also improving our communication processes to ensure timely and accurate customer updates.