Rudderstack Outage History

RudderStack has cleared all backlog across all the python transformations. We do not see any lingering impact to customers. If you have questions contact your account team

RudderStack has cleared all backlog across all customers. We do not see any lingering impact to customers. If you have questions contact your account team

We experienced an issue with our AWS cloud provider, which caused temporary disruptions to web app login functionality. https://health.aws.amazon.com/health/status Users may have been unable to log in during this period. Ingestion services were not impacted, all data continued to flow and process as expected.

The incident has been resolved. Changes made to determine how event_source is set to TikTok events broke the deliveries for its server SDK sources. Change has been reverted and failures have been replayed for all customers with configured backup

The incident has been resolved. Recently, Salesforce decided to enforce client_id validation and rolled out changes that made this field mandatory. Fix has been made pass the client_id. Failures have been replayed for all customers with salesforce destinations. Pipelines with rETL sources to salesforce will be processed in the next scheduled sync

Deliveries to Google Ads Offline Conversions were impacted due to issues (unavailable, high latency and unexpected behavior) with Google Ads API as reported by Google (https://ads.google.com/status/publisher/incidents/xF9wKox7evao5WupR7mu). We’ve identified the affected customers and have completed a replay for all impacted destinations.

This incident has been resolved.

An issue was identified with the delivery module for S3 & S3_DATALAKE destinations, leading to the incorrect duplication of the prefix in the final file path. This resulted in data being uploaded to an unintended location, making it unavailable for any downstream consumption for the duration of the incident. The flag was enabled only for free tier and starter and growth customers. Follow the below recommended steps to recover the data: Prerequisite: Make sure the AWS CLI is installed and configured locally to access the relevant S3 bucket. 1. Go to the Rudderstack configuration page of your S3/S3 Datalake destinations and copy bucket name and prefix. 2. Copy all the files from the double-prefix path to the single prefix path. Command: aws s3 cp s3://{bucket_name}/{prefix}/{prefix}/ s3://{bucket_name}/{prefix}/ --recursive 3. Delete the original files (optional) Command:aws s3 rm s3://{bucket_name}/{prefix}/{prefix}/ --recursive

This incident has been resolved and no further ingestion failures are being observed

This incident has been resolved. During the migration of our internal database, certain custom configurations from the previous version were not restored in the new setup. This led to degraded service performance, causing failing health checks and an increase in service restarts.

On 18th Feb, We experienced a partial outage affecting some sources: Webhook, Customer.io, AppsFlyer, Shopify, Braze, Auth0, Iterable, Monday, Adjust, Pipedream, and Segment. This issue impacted some Enterprise US customers, leading to failure to ingest data. Our team identified and resolved the issue, all affected services are fully operational. Data is now flowing as expected. Start Time: 08:55 UTC End Time: 09:30 UTC

On 5th feb, Events began failing for Google Ads offline conversion destinations due to the deprecation of Google Ads API v16 in both the US and EU regions. As a result, affected customers may have experienced missing or delayed data. We performed an event replay for impacted destinations to recover lost data. Customers who did not have a backup in place may still experience data loss. The incident is now fully resolved. If you continue to see any related issues or have any concerns, please contact our support team for further assistance.

Ingestion is back to normal and this incident is resolved

Incident Summary: RudderStack encountered an issue where JavaScript SDK plugin artifacts were served with incorrect MIME type headers (binary/octet-stream) from our CDN. This caused modern browsers to reject the scripts, resulting in plugin loading failures and a loss of instrumented events to RudderStack and device mode destinations. Impact: The issue did not affect all customers uniformly due to the nature of content distribution via the CDN. Some customers experienced the issue earlier as their regions picked up the changes, while others saw delayed or minimal impact. Resolution: The incorrect MIME type was promptly corrected to application/javascript, and normal functionality was restored. We sincerely apologize for any inconvenience caused and thank you for your understanding. If you have further questions or concerns, please contact our support team.

Customers in EU region on pro tier ingestion got affected from 10:00AM to 10:15AM UTC. For JS and mobile SDKs, events would have got retried and server-side SDKs, there is a possibility of data loss. This is resolved now.