Rudderstack experienced a minor incident on May 28, 2025 affecting Data Processing Pipeline (US) and Warehouses, lasting 2h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 29, 2025, 01:40 PM UTC
Issue has been identified with S3 and S3 datalake deliveries. Actively working on the fix.
- resolved May 29, 2025, 01:40 PM UTC
An issue was identified with the delivery module for S3 & S3_DATALAKE destinations, leading to the incorrect duplication of the prefix in the final file path. This resulted in data being uploaded to an unintended location, making it unavailable for any downstream consumption for the duration of the incident. The flag was enabled only for free tier and starter and growth customers. Follow the below recommended steps to recover the data: Prerequisite: Make sure the AWS CLI is installed and configured locally to access the relevant S3 bucket. 1. Go to the Rudderstack configuration page of your S3/S3 Datalake destinations and copy bucket name and prefix. 2. Copy all the files from the double-prefix path to the single prefix path. Command: aws s3 cp s3://{bucket_name}/{prefix}/{prefix}/ s3://{bucket_name}/{prefix}/ --recursive 3. Delete the original files (optional) Command:aws s3 rm s3://{bucket_name}/{prefix}/{prefix}/ --recursive