Lightstep experienced a major incident on February 16, 2023 affecting US - Trace Assembly and US - Trace Statistics, lasting 1h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 16, 2023, 04:39 PM UTC
We are currently investigating an issue with span ingest into: ingest.lightstep.com.
- investigating Feb 16, 2023, 05:05 PM UTC
We have identified an issue ingesting spans for customers sending traffic directly to ingest.lightstep.com. Customers that are running microsatellites are not affected.
- identified Feb 16, 2023, 05:05 PM UTC
A suspected issue has been identified and a fix is being rolled out.
- identified Feb 16, 2023, 05:06 PM UTC
We have identified an issue ingesting spans for customers sending traffic directly to ingest.lightstep.com. Customers that are running microsatellites are not affected.
- identified Feb 16, 2023, 05:31 PM UTC
Span ingestion appears to be functional. Trace assembly is still impacted and actively being worked on.
- identified Feb 16, 2023, 05:35 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Feb 16, 2023, 05:43 PM UTC
We have deployed the fixes and are actively monitoring the affected systems.
- resolved Feb 16, 2023, 05:50 PM UTC
All affected systems appear to be returned to a working state.