Opus Interactive incident

Disk Latency / Network Latency

Major Resolved View vendor source →

Opus Interactive experienced a major incident on July 11, 2019 affecting PDX1 Network and HIO1 Network and 1 more component, lasting 12h 10m. The incident has been resolved; the full update timeline is below.

Started
Jul 11, 2019, 11:42 PM UTC
Resolved
Jul 12, 2019, 11:53 AM UTC
Duration
12h 10m
Detected by Pingoru
Jul 11, 2019, 11:42 PM UTC

Affected components

PDX1 NetworkHIO1 NetworkPDX1 Core ServicesHIO1 Core Services

Update timeline

  1. identified Jul 11, 2019, 11:42 PM UTC

    The issue is being investigated. The issue is isolated to a specific cluster. We are migrating everyone off to improve functionality and performance. Further action is anticipated. Expect sluggish network connectivity and dropped packets until the issue is resolved.

  2. identified Jul 12, 2019, 01:15 AM UTC

    The migrations are moving forward. As the datastore becomes more clear the migrations will likely move faster and performance will improve for everyone.

  3. identified Jul 12, 2019, 02:20 AM UTC

    We've made good progress and see positive results in the bulk of the impacted environment. VM's are continuing to move with steadily increasing speed.

  4. identified Jul 12, 2019, 04:47 AM UTC

    More than half the migrations are complete. We continue to see improved performance.

  5. resolved Jul 12, 2019, 11:53 AM UTC

    This incident has been resolved.