Smarty incident

Single IP address caused some traffic to be null routed

Notice Resolved View vendor source →

Smarty experienced a notice incident on November 28, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
Nov 28, 2024, 05:00 PM UTC
Resolved
Nov 28, 2024, 05:00 PM UTC
Duration
—
Detected by Pingoru
Nov 28, 2024, 05:00 PM UTC

Update timeline

  1. resolved Dec 09, 2024, 07:45 PM UTC

    Around 15:00 (3pm) UTC on 2024 November 28, a single IP address at one of our datacenters experienced a routing issue that caused some requests to be "null routed", which may have resulted in some clients experiencing 500 level errors during that time. This issue persisted until 17:00 (5pm) UTC on 2024 November 28, at which time our upstream provider resolved the issue. Our recommendation to clients to mitigate an issue such as this in the future is to maintain a pool of active TCP connections to the server nodes and to only send traffic over active, healthy TCP connections. Then a background "thread" (or programmatic equivalent) could attempt to connect to the suspect IPv4 address. Once that connection has determined to be healthy, regular traffic can be resumed over that TCP connection.