[US] Internet Service Provider Outage

Update timeline

1. identified Sep 25, 2024, 09:31 PM UTC

   Hello lovelies! It seems our ISP is experiencing an outage once again. We’ve reached out to AT&T, and here’s the response we received: "Hi, it's AT&T. There's an outage in your area so your appointment is on hold. We'll contact you once the outage is over." We’ll provide updates as soon as we have more information. Thank you for your patience and understanding! ❤️ Best regards, The Astro's Lounge Staff Team

2. identified Sep 26, 2024, 04:36 AM UTC

   We have received an update from our ISP regarding the ongoing outage. The issue stems from a fault in a recently installed Fiber Distribution Hub that bridges connections for businesses and homeowners in the affected area. This fault has caused a total internet blackout in the region, and as a result, our fallback bridge connection—which normally takes over if the primary fiber source goes offline—was also impacted due to the Hub failure. Since this location is our primary hub for hosting game servers, DNS, and APIs, the blackout has caused these services to become unreachable. With our primary DNS offline, worker locations are unable to communicate with our main database, leading to worker 500 errors and game servers appearing offline. Unfortunately, this means that our services will remain inaccessible within the United States until the issue is fully resolved. Our ISP is working as quickly as possible to restore services, with the latest estimated restoration time being 5 PM CDT. Thank you for your patience and understanding during this outage. Best regards, The Astro's Lounge Staff Team

3. monitoring Sep 26, 2024, 06:29 PM UTC

   Connection has been successfully restored to our primary data center. We are now in the process of conducting network tests to ensure stability and performance. All services are expected to be fully operational within the next hour. Thank you for your patience, and we will provide further updates if needed. Best regards, The Astro's Lounge Staff Team

4. resolved Sep 26, 2024, 07:22 PM UTC

   All servers are now online and fully operational. Our databases and cache servers have successfully finished syncing data, allowing you to resume playing as before. We sincerely apologize for the inconvenience caused by this outage. While the issue was beyond our control, we worked tirelessly to mitigate the impact. This was the longest outage Astro's Lounge has ever experienced in our years of hosting servers, and we greatly appreciated your patience and understanding during this time. As a gesture of goodwill, if you are currently renting a server from us, we will be providing a one-month credit to make up for the downtime. Thank you for your understanding, The Astro's Lounge Staff Team