LogDNA incident

For some customers, newly submitted logs are only appearing in the UI after long delays

Minor Resolved View vendor source →

LogDNA experienced a minor incident on March 24, 2021 affecting Search and Alerting and 1 more component, lasting 4d 14h. The incident has been resolved; the full update timeline is below.

Started
Mar 24, 2021, 10:38 PM UTC
Resolved
Mar 29, 2021, 01:11 PM UTC
Duration
4d 14h
Detected by Pingoru
Mar 24, 2021, 10:38 PM UTC

Affected components

SearchAlertingLivetailArchiving

Update timeline

  1. identified Mar 24, 2021, 10:38 PM UTC

    For some customers, newly submitted logs are being ingested successfully but are only appearing in our UI after long delays. Our engineers have identified the cause and are working towards a solution.

  2. identified Mar 24, 2021, 11:44 PM UTC

    Delays are still being experienced by some customers. We continue to work towards a solution.

  3. identified Mar 25, 2021, 01:54 AM UTC

    New logs are still appearing in the UI with delays for some customers. We are continuing to investigate.

  4. identified Mar 25, 2021, 07:07 AM UTC

    We are still experiencing some delays, new logs are continuing to be processed. Investigations are ongoing.

  5. identified Mar 25, 2021, 08:43 PM UTC

    We are still experiencing some delays, new logs are continuing to be processed and our engineers are taking steps to mitigate the impact.

  6. identified Mar 26, 2021, 05:01 AM UTC

    As we continue working on this issue, we have temporarily scaled down livetail and alerting. Some customers may not see live tail and receive new alerts. New logs still appear in the UI with delays.

  7. identified Mar 26, 2021, 10:33 AM UTC

    We have resumed live tail and alerting now. Some users may still experience delays with searching and archiving. We are continuing to monitor the situation.

  8. identified Mar 26, 2021, 03:45 PM UTC

    As we continue working on this issue, we have temporarily scaled down live-tail and alerting. Some customers may not see live tail and receive new alerts. New logs still appear in the UI with delays.

  9. identified Mar 28, 2021, 06:48 PM UTC

    We have resumed all services and functionality. We are continuing to monitor the situation.

  10. resolved Mar 29, 2021, 01:11 PM UTC

    This incident has been resolved.