Smarty incident

Intermittent HTTP 503 Errors:

Notice Resolved View vendor source →

Smarty experienced a notice incident on December 30, 2024 affecting US East 1 and US East 1 and 1 more component, lasting —. The incident has been resolved; the full update timeline is below.

Started
Dec 30, 2024, 10:32 PM UTC
Resolved
Dec 30, 2024, 10:32 PM UTC
Duration
—
Detected by Pingoru
Dec 30, 2024, 10:32 PM UTC

Affected components

US East 1US East 1US West 1US West 1US Central 1US Central 1US East 1US East 1US West 1US West 1

Update timeline

  1. resolved Dec 30, 2024, 10:32 PM UTC

    During a regular upgrade of our control plane, several controller nodes became unresponsive. This resulted in various routing issues which were manifest by intermittent HTTP 503 errors visible in the HTTP response to some callers. Our monitoring systems alerted our Operations Engineering Team of the anomaly and, as a result, the affected cluster, located in our US Central region (Chicago), was immediately removed from production rotation. The affected cluster has been restored to full health by provisioning new cloud resources using our IaC (infrastructure as code) systems. Following this, the cluster was confirmed to be was healthy via our automated monitoring and was subsequently re-introduced into active rotation.