Ryver incident

Unable to connect and/or 502 Bad Gateway

Minor Resolved View vendor source →

Ryver experienced a minor incident on February 10, 2017, lasting 2h 54m. The incident has been resolved; the full update timeline is below.

Started
Feb 10, 2017, 02:20 AM UTC
Resolved
Feb 10, 2017, 05:14 AM UTC
Duration
2h 54m
Detected by Pingoru
Feb 10, 2017, 02:20 AM UTC

Update timeline

  1. investigating Feb 10, 2017, 02:20 AM UTC

    We are currently investigating this issue.

  2. investigating Feb 10, 2017, 02:39 AM UTC

    Amazon reported an issue with US-West-1 where the Ryver data center is located. Monitoring and will provide more details.

  3. monitoring Feb 10, 2017, 03:12 AM UTC

    Here is the information we have from Amazon so far: Event: EC2 VPC network health intra AvailabilityZone issue Status: Open Region/Availability Zone: us-west-1 (Ryver's servers are in this zone) Start time: February 9, 2017 at 6:58:00 PM UTC-7 End time: Ongoing

  4. monitoring Feb 10, 2017, 03:18 AM UTC

    There has been progress on the networking issues. Update from Amazon: We have identified the root cause of the network connectivity issues for instances and failures of newly launched instances in a single Availability Zone in the US-WEST-1 Region. Connectivity to some instances has been restored and we continue to work on the remaining instances.

  5. monitoring Feb 10, 2017, 04:00 AM UTC

    We are seeing a full return of connectivity between our servers. We continue to monitor.

  6. resolved Feb 10, 2017, 05:14 AM UTC

    This incident has been resolved.

  7. postmortem Aug 03, 2018, 05:33 PM UTC

    Here is the final log we received on Amazon for Northern California Data Center: 5:28 PM PST We are investigating network connectivity issues in a single Availability Zone in the US-WEST-1 Region. 6:15 PM PST We continue to investigate network connectivity issues for instances and failures of newly launched instances in a single Availability Zone in the US-WEST-1 Region. 7:09 PM PST We have identified the root cause of the network connectivity issues for instances and failures of newly launched instances in a single Availability Zone in the US-WEST-1 Region. Connectivity to some instances has been restored and we continue to work on the remaining instances. 8:10 PM PST Between 4:52 PM and 7:49 PM PST we experienced network connectivity issues for instances and failures of newly launched instances in a single Availability Zone in the US-WEST-1 Region. The issue has been resolved and the service is operating normally. We are having a meeting today with our Amazon AWS representative to discuss the cause for this problem, and how we can best avoid being impact by such an outage in the future.