Thunderstore experienced a minor incident on April 7, 2021 affecting Thunderstore API and Thunderstore Website, lasting 10d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 07, 2021, 05:58 PM UTC
We've been investigating an issue that has lead to abrupt downtime of the website and API since 2021-04-06. The investigation is still ongoing, and the service may have degraded performance until the root cause has been identified and fully resolved.
- monitoring Apr 18, 2021, 03:06 AM UTC
We believe to have identifier the root cause of the issue, and applied fixes that address the issue. In more technical details, each request passed to the application server spawns it's own short lived database connection to our database, through a connection pooler. Since our database is using a DNS name, this means for every request, the request handler has to perform a DNS query for resolving the database address. During high traffic, we started to see DNS queries for resolving the database address fail, thus causing those requests to also go unhandled. This lead to clients (e.g. r2modman) re-trying the requests, and thus turning into even greater amounts of traffic. Additionally, since we are using kubernetes, each DNS lookup actually attempted to resolve the cluster-local names first before looking up external sources. In practice this meant that for every database connection, we'd perform 3 invalid DNS queries first, until on the 4th query we'd look from the correct (external) servers. The DNS queries work over UDP, so our current theory is that during high traffic, some of those UDP packets were lost in transit. Normally this wouldn't happen with the amount of traffic we were seeing, but due to inefficient application architecture there was substantially higher amounts of in-cluster networking happening when serving responses. For reference, the traffic was high enough to fully saturate a server node's network connection in some cases. To mitigate the issue, we have: - Ensured our cluster-local DNS cache is working - Eliminated the extra DNS queries from database address lookup - Set up a more efficient HTTP cache in front of the API With these measures in place, the issue seems to be resolved.
- resolved Apr 18, 2021, 03:06 AM UTC
This incident has been resolved.