Engine Yard incident

Cloud Dashboard currently not working

Minor Resolved View vendor source →

Engine Yard experienced a minor incident on October 21, 2016, lasting 2h 22m. The incident has been resolved; the full update timeline is below.

Started
Oct 21, 2016, 11:40 AM UTC
Resolved
Oct 21, 2016, 02:02 PM UTC
Duration
2h 22m
Detected by Pingoru
Oct 21, 2016, 11:40 AM UTC

Update timeline

  1. investigating Oct 21, 2016, 11:40 AM UTC

    We are currently seeing failures to load the Cloud Dashboard (https://cloud.engineyard.com/). Platform engineers are currently investigating the issue.

  2. identified Oct 21, 2016, 11:59 AM UTC

    We have identified an issue with networking at AWS and continue to investigate.

  3. identified Oct 21, 2016, 12:17 PM UTC

    The networking issues are also affecting our ZenDesk support portal (https://support.cloud.engineyard.com) so customers may find they are unable to submit tickets at this time. Please contact support via irc.freenode.net channel #engineyard for urgent support if required.

  4. identified Oct 21, 2016, 12:36 PM UTC

    AWs has identified the root cause of the issue causing errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region, and are currently working to resolve.

  5. identified Oct 21, 2016, 01:04 PM UTC

    AWS are working to resolve the DNS resolution failures at this time (http://status.aws.amazon.com/).

  6. monitoring Oct 21, 2016, 01:19 PM UTC

    AWS continue to work to resolve the issue and we are now seeing the Cloud Dashboard and ZenDesk support portal back online. We are yet to receive an official all clear, so please monitor the situation and contact us via the support board or IRC should you require further assistance.

  7. resolved Oct 21, 2016, 02:02 PM UTC

    6:36 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.