Datadog Govcloud incident

Logs query results returning unexpected results for some queries and monitors

Major Resolved View vendor source →

Datadog Govcloud experienced a major incident on October 4, 2023 affecting Log Management and Monitors, lasting 4h 29m. The incident has been resolved; the full update timeline is below.

Started
Oct 04, 2023, 10:04 AM UTC
Resolved
Oct 04, 2023, 02:33 PM UTC
Duration
4h 29m
Detected by Pingoru
Oct 04, 2023, 10:04 AM UTC

Affected components

Log ManagementMonitors

Update timeline

  1. identified Oct 04, 2023, 10:04 AM UTC

    We have identified an issue for querying on logs timestamped between 2023-10-02 16:40 UTC and 2023-10-03 20:40 UTC. A fix for this issue is currently being tested. All queries on logs with timestamps from outside of this timeframe are working as expected.

  2. identified Oct 04, 2023, 11:28 AM UTC

    We're still working on a fix for historical data impacted by this incident.

  3. identified Oct 04, 2023, 12:19 PM UTC

    We're still working on a fix for historical data impacted by this incident.

  4. identified Oct 04, 2023, 01:05 PM UTC

    We have successfully tested a fix for this issue and are currently deploying it to resolve this incident.

  5. identified Oct 04, 2023, 01:15 PM UTC

    Fix has been rolled out and we are currently monitoring to confirm full resolution.

  6. monitoring Oct 04, 2023, 02:05 PM UTC

    A fix has been implemented and we are monitoring the results.

  7. resolved Oct 04, 2023, 02:33 PM UTC

    This incident has been resolved.