CloudCannon experienced a minor incident on May 4, 2026 affecting App (app.cloudcannon.com), lasting 14h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 04, 2026, 02:07 PM UTC
We are currently investigating this issue.
- identified May 04, 2026, 02:38 PM UTC
The issue has been identified and a fix is being implemented.
- identified May 04, 2026, 03:12 PM UTC
Certain sites are stuck trying to sync at the start of a build. This is not as widespread as initially expected. We will continue our investigation during business hours. Thank you for your patience.
- identified May 04, 2026, 10:11 PM UTC
We have identified the issue and we are working on a fix. Sites will be rebuilt when resolved.
- identified May 05, 2026, 12:30 AM UTC
The issue is patched and we are working through the recovery process.
- identified May 05, 2026, 04:07 AM UTC
Recovery is complete. We are working to rebuild sites with a broken build.
- resolved May 05, 2026, 04:39 AM UTC
All files have been restored and sites are rebuilt. Thanks for your patience surrounding this issue. If you experience any further issues, please contact support.
- postmortem May 05, 2026, 09:04 PM UTC
## Postmortem: S3 Storage Incident \(May 3–5, 2026\) ### Root Cause Between May 3–5, 2026, a bug in CloudCannon's site-deletion logic caused a background job to delete a large portion of files from our production S3 bucket. When a site is deleted, a background job removes its associated files from S3 using the site's storage prefix as the deletion target. The issue was caused when a site with a blank storage prefix — a value that should not be permitted but lacked validation — was deleted. The deletion function had no guard against a blank prefix, so it resolved to match every object in the bucket. The job's automatic retry behavior meant that when it timed out mid-deletion, it resumed on each retry, progressively deleting further through the bucket alphabetically, and the loss of source files caused builds to fail. All affected files have been fully recovered using S3 versioning. ### Impact * Sites whose storage prefix fell alphabetically before the job's timeout point lost all source files in S3. * All builds for affected sites failed. * Syncing diffs were also deleted and were unavailable during this window. These have been recovered. * Deployed sites, served from a separate bucket, were not affected. * Hosting was relatively unaffected due to caching layers. * No customer data was permanently lost — S3 versioning was enabled, so all files were recoverable. ### Resolution Files were recovered by removing the S3 delete markers created by the bulk deletion, restoring all objects in place. No database changes or manual re-syncing were required. The following fixes have been shipped: * The S3 deletion function now returns immediately if given a blank prefix. * The site deletion job now validates the storage prefix before running. * The site model now enforces a non-blank storage prefix at the data layer.