TCPShield Outage History

Traffic across all services in Tokyo has remained stable since the replacement. We are marking this incident as resolved.

The spare optic has been enabled and traffic has been restored to the location as of 04:12 UTC. Traffic previously on Tokyo has moved back to Seoul.

The previously reported latency increase has resolved as of 16:46 UTC following re-convergence on the backbone. We await root cause from the network team.

A long running TCP flood caused connection state tables to fill in some locations, resulting in intermittent login and ping failures. The root cause of the excess state allocations has been identified and fully remediated as of 01:23 UTC. All regions are operating normally and we will continue to monitor closely. If you are still experiencing any issues, please open a support ticket and our staff will be happy to assist.

Between 07:43 and 07:46 UTC, our Frankfurt network experienced intermittent packet loss due to attack traffic leakage. The root cause was quickly identified as a single destination IP had been configured with a weaker mitigation profile, allowing a portion of attack volume to bypass filtering. The affected IP was reconfigured immediately. As a precaution, we also completed a full audit of all public-facing IPs to ensure consistent mitigation posture across the fleet. No further impact is expected.

Following hardware replacement completed by our datacenter, service in Tokyo has been fully restored. All player traffic has been routed back to Tokyo as of July 7th, 14:00 UTC, and the location is operating normally. We thank you for your patience while replacement hardware was sourced and installed. データセンターによるハードウェア交換が完了し、東京拠点のサービスは完全に復旧いたしました。現在、すべてのトラフィックは東京経由で正常にルーティングされています。

This incident has been resolved.

From 09:21 UTC to 09:43 UTC, the GSL network team performed routine maintenance on the scrubbing hardware in Frankfurt for the purpose of an OS upgrade + NIC swap. Normally, these procedures do not cause any form of traffic blackholing in other sites. Unfortunately for us, the way our statics are setup in Frankfurt, such procedures did cause a full blackhole in traffic for the duration of the maintenance. We've stressed to the team the importance of heads up in cases such as these, and informed them on the necessity on timing it closer to off peak windows. No further maintenance is required in Frankfurt at this time.

This incident has been resolved.

This incident has been resolved.

At 19:42:10 UTC, our Amsterdam location experienced forwarding degradation lasting approximately 1 minute. The root cause is being investigated by GSL at this time, as the issue is external to our own infrastructure. We will share updates as we learn more, and will pull routing from Amsterdam to alternate locations in the event of a recurrence. In the meantime, we continue to monitor.

From 19:41 to 20:12 UTC, all European locations experienced significant packet loss affecting all European backends. The root cause was backbone link saturation triggered by an attack targeting an adjacent customer on GSL's network, which caused spillover effects across our infrastructure and GSL's network more broadly. In response, the GSL team (myself included) has placed this customer into a preconfigured mitigation stance that will proactively block this traffic upon re-occurrence. We do not anticipate a repeat of this event.

Login error rate has been restored and a corrective patch has been deployed, tested, and verified globally throughout the network. We continue to monitor and do not expect this issue to resurface. If you see any issues, please don't hesitate to reach out via ticket.

PSU replacement was successful. Traffic has re-converged and we will continue to monitor for anything unusual.

Routes have successfully reconverged, latency has also normalized.

The degradation has been resolved and continuing to monitor.

At around 1:48 UTC, our network globally received a series of sophisticated TCP attacks aiming to replicate legitimate traffic. These attacks were executed at high packet rate, peaking as high as 3Gpps (billion packets per second) measured from the borders. By 1:55 UTC, we understood the full scope of the bypassing nature of the attack. By 03:38 UTC, tests have been written to match the corresponding mitigation to the new attack vector. By 03:45 UTC, the patch has been deployed globally to the network. And finally, at 05:20 UTC, we received 2 hits of the same attack vector with no further impact to our network. We continue to record and monitor all network flows towards our border and remain available to act quickly should any additional attempts arise.

Datacenter confirmed fault. Services restored as of 09:52 UTC.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

From 17:00 to 18:00 UTC, Vint Hill saw a series of performance and latency degradations affecting some traffic in the region. The root cause has been identified and fully mitigated. This is a retrospective notification regarding the incident that occurred earlier today, and we continue to monitor stability.

The issue has not re-surfaced since corrective changes to these regions.

From 18:00 to 18:15 UTC, state tables filled in Frankfurt resulting in new logins being unable to be processed. We have increased table capacity and this has since been resolved.