Hydrow incident

live rows not running properly

Major Resolved View vendor source →

Hydrow experienced a major incident on June 13, 2023 affecting Hydrow Service, lasting 2h 2m. The incident has been resolved; the full update timeline is below.

Started
Jun 13, 2023, 08:10 PM UTC
Resolved
Jun 13, 2023, 10:12 PM UTC
Duration
2h 2m
Detected by Pingoru
Jun 13, 2023, 08:10 PM UTC

Affected components

Hydrow Service

Update timeline

  1. investigating Jun 13, 2023, 08:10 PM UTC

    We have reports and logging that show a problem with live rows happening this afternoon. We are investigating.

  2. investigating Jun 13, 2023, 08:41 PM UTC

    The 4PM Eastern Live Row was not able to start or end the timer, or record any meters. We apologize for the difficulty. We are not aware of any other impacts to our customers; only live rows appear to be affected. We believe the issue is caused by an outage acknowledged by one of our vendors, AWS. We are monitoring that issue, and waiting for them to resolve it.

  3. monitoring Jun 13, 2023, 09:28 PM UTC

    AWS reports that the issue is on its way to resolution. We believe the live row for 6PM Eastern will be successful. We continue to monitor closely.

  4. resolved Jun 13, 2023, 10:12 PM UTC

    Our 6pm Eastern live row is underway with no known issues and we do not anticipate further disruption. We will continuing to monitor for any further problems.