Bigeye incident

Some missed metric runs

Minor Resolved View vendor source →

Bigeye experienced a minor incident on May 9, 2023 affecting API and UI, lasting 5d 22h. The incident has been resolved; the full update timeline is below.

Started
May 09, 2023, 06:47 PM UTC
Resolved
May 15, 2023, 05:08 PM UTC
Duration
5d 22h
Detected by Pingoru
May 09, 2023, 06:47 PM UTC

Affected components

APIUI

Update timeline

  1. investigating May 09, 2023, 06:47 PM UTC

    We are experiencing occasional missed metric runs. Engineers are investigating the root cause. Users of the web app and API are not affected. Next update: 2 hours

  2. investigating May 09, 2023, 10:32 PM UTC

    We have identified a potential root cause for the metrics runs issue, and have a hot fix underway.

  3. identified May 10, 2023, 04:48 PM UTC

    We have identified the problem and are working on fixing the root cause

  4. monitoring May 10, 2023, 10:18 PM UTC

    A fix has been implemented and we are monitoring the results

  5. resolved May 15, 2023, 05:08 PM UTC

    After monitoring for the weekend we are confident that our fixes have resolved the problem