brainCloud incident

We are investigating reports of issues with launching room and relay servers.

Minor Resolved View vendor source →

brainCloud experienced a minor incident on September 12, 2024 affecting Hosting Services, lasting 2h 8m. The incident has been resolved; the full update timeline is below.

Started
Sep 12, 2024, 04:12 PM UTC
Resolved
Sep 12, 2024, 06:20 PM UTC
Duration
2h 8m
Detected by Pingoru
Sep 12, 2024, 04:12 PM UTC

Affected components

Hosting Services

Update timeline

  1. investigating Sep 12, 2024, 04:12 PM UTC

    We are investing some odd metrics in our hosting services.

  2. investigating Sep 12, 2024, 04:46 PM UTC

    The issue may be related to connectivity issues with the certain EU regions. We are continuing to investigate...

  3. identified Sep 12, 2024, 05:22 PM UTC

    The issue is that hosted servers in the AWS eu-central-1 (Frankfurt) region are launching with connectivity issues. The issues we are seeing are "Instance reachability check failed" and "Attached EBS reachability check failed". GameDevs could temporarily turn off this region in their room server configurations to avoid the issue. We are continuing to monitor.

  4. identified Sep 12, 2024, 05:51 PM UTC

    For now we are removing the troublesome subnet (1a) from the eu-west-1 environment.

  5. resolved Sep 12, 2024, 06:20 PM UTC

    All metrics look good again. We have contacted Amazon about the issue. We will return that availability zone to service once we get an assessment from them (and assurances that things are now fixed).