Datadog Integration incident

VictorOps Elevated API Errors

Minor Resolved View vendor source →

Datadog Integration experienced a minor incident on September 10, 2024 affecting AWS and Victorops, lasting 2h 6m. The incident has been resolved; the full update timeline is below.

Started
Sep 10, 2024, 02:06 AM UTC
Resolved
Sep 10, 2024, 04:12 AM UTC
Duration
2h 6m
Detected by Pingoru
Sep 10, 2024, 02:06 AM UTC

Affected components

AWSVictorops

Update timeline

  1. investigating Sep 10, 2024, 02:06 AM UTC

    We're experiencing an elevated level of API errors sending notifications to VictorOps and are currently looking into the issue. As a result customers may not receive monitor notifications that use the VictorOps integration.

  2. investigating Sep 10, 2024, 03:24 AM UTC

    We are continuing to investigate this issue.

  3. monitoring Sep 10, 2024, 03:30 AM UTC

    We are no longer seeing elevated error rates and will continue to monitor the VictorOps API to ensure stability. We have notified the VictorOps support team of the issue.

  4. resolved Sep 10, 2024, 04:12 AM UTC

    The VictorOps API is not longer showing elevated error rates. Any notifications that previously failed due to API errors have been re-send to VictorOps.