LaunchDarkly Outage History

This incident has been resolved. Between 12:00 and 12:12 PM PDT on August 26, server-side SDKs establishing new streaming connections experienced elevated error rates. Existing streaming connections were not affected, and polling requests continued to be served with error rates below 1%. The issue was caused by an internal configuration change, which has been reverted. All systems have recovered and are operating normally. No customer data was lost or compromised.

Email delivery has been restored as of 2:32 PM UTC. Sign-up verification, password reset, teammate invitation, and notification emails are sending normally again. We are re-sending sign-up verification and password reset emails that failed to deliver during the incident window (August 22, 11:48 PM CDT through today). If you attempted to sign up or reset a password during that period and haven't received an email within the next few hours, please retry the action or contact support. We are continuing to monitor email delivery closely and will post a final update once we've confirmed sustained recovery.

This incident has been resolved.

The issue is being marked as resolved. We're seeing improved latencies and our systems have recovered.

SDKs restored to operational

The issue with Observability Traces has been resolved. Missing trace data has been recovered.

The issue with session replay and error ingest has been resolved. We've caught up on all backlogged data as of 1:01pm PT.

Between 11:32 and 11:48 PT, customers experienced increased latency and elevated error rates on API endpoints. Latency and error rates have returned to normal, and the incident has been resolved.

All service availability has been fully restored. Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it. To determine if an application needs to be restarted, customers can look at their application logs. Affected SDKs will emit (depending on language) logs containing one of "giving up permanently", "Invalid SDK key", "unauthorized", or "not authorized", or "401". Services which emitted logs such as these before 9:13AM PT may be unable to receive flag updates and will be unable to send events to LaunchDarkly until they are restarted.

LaunchDarkly UI has recovered. We will continue to monitor Flag Delivery and event ingest here: https://status.launchdarkly.com/incidents/chywwz01ptb0.

This incident has been resolved.

Event ingestion is operating normally. Experimentation results, Guarded Rollouts, and in-product data charts were delayed ~5 min between 7:13–7:53 AM PT. No data was lost. We apologize for any inconvenience.

All error rates have normalized.

This incident has been resolved.

This incident has been resolved.

We're now seeing restored access in the affected regions.

The issue with Observability data ingest for OpenTelemetry data has been resolved. Impacted services have returned to normal operation.

Monitoring has confirmed no further impact outside of the initially reported 7:30-7:37pm PT timeframe

The incident has been resolved and the flag dashboard is successfully loading.

We have confirmed Slack notifications are now coming through for segment approvals again.

This incident has been resolved. Event ingestion is operating normally and all Experimentation results and Guarded Rollouts are fully restored. No data was lost. We apologize for any inconvenience caused.

All queued events have been processed and data ingestion is operating normally. No data was lost during this incident.

We've implemented a fix and have mitigated impact as of 9:25pm PT. Streaming SDKs should be able to connect successfully at this time.