Lightstep incident

Lightstep experience major issues with web UI, query and ingest.

Critical Resolved View vendor source →

Lightstep experienced a critical incident on July 11, 2023 affecting US - Trace Assembly and PagerDuty Events API and 1 more component, lasting 9h 27m. The incident has been resolved; the full update timeline is below.

Started
Jul 11, 2023, 03:21 AM UTC
Resolved
Jul 11, 2023, 12:48 PM UTC
Duration
9h 27m
Detected by Pingoru
Jul 11, 2023, 03:21 AM UTC

Affected components

US - Trace AssemblyPagerDuty Events APIUS - Service DirectoryUS - Unified AlertingUS - DashboardsUS - Streams AlertingUS - Trace StatisticsSlack Apps/IntegrationsUS - NotebooksUS - Metrics

Update timeline

  1. investigating Jul 11, 2023, 01:20 AM UTC

    We are currently investigating this issue.

  2. investigating Jul 11, 2023, 01:46 AM UTC

    We are continuing to investigate this issue.

  3. investigating Jul 11, 2023, 02:09 AM UTC

    We are continuing to investigate this issue.

  4. investigating Jul 11, 2023, 02:10 AM UTC

    Ingest of all data is currently recovering. The web UI is back up. Queries should be working.

  5. investigating Jul 11, 2023, 03:21 AM UTC

    Most services should be operating normal. There may still be issues with alerting. We continue to investigate.

  6. investigating Jul 11, 2023, 04:31 AM UTC

    We are continuing to investigate this issue. There has been a regression.

  7. monitoring Jul 11, 2023, 05:12 AM UTC

    A fix has been implemented and we are monitoring the results.

  8. resolved Jul 11, 2023, 12:48 PM UTC

    This incident has been resolved.