Waterford incident

Degraded performance when starting courses

Minor Resolved View vendor source →

Waterford experienced a minor incident on September 12, 2023 affecting my.waterford.org and studentportal.waterford.org, lasting 3h 17m. The incident has been resolved; the full update timeline is below.

Started
Sep 12, 2023, 03:08 PM UTC
Resolved
Sep 12, 2023, 06:26 PM UTC
Duration
3h 17m
Detected by Pingoru
Sep 12, 2023, 03:08 PM UTC

Affected components

my.waterford.orgstudentportal.waterford.org

Update timeline

  1. investigating Sep 12, 2023, 03:08 PM UTC

    We have identified longer than usual wait times when launching and playing activities and are actively investigating the issue.

  2. investigating Sep 12, 2023, 04:34 PM UTC

    We have identified the root cause and are working on a mitigation strategy.

  3. identified Sep 12, 2023, 04:34 PM UTC

    The issue has been identified and a fix is being implemented.

  4. monitoring Sep 12, 2023, 05:34 PM UTC

    Additional mitigation steps have been taken, and wait times are back to normal levels.

  5. resolved Sep 12, 2023, 06:26 PM UTC

    The issue has been resolved however, it impacted our data replication processes reports, dashboards and upstart assignments may be delayed as jobs resume normal operation.