Svix incident

Elevated response times in the US region for a small percentage of the requests

Minor Resolved View vendor source →

Svix experienced a minor incident on August 24, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Aug 24, 2023, 04:00 PM UTC
Resolved
Aug 24, 2023, 04:00 PM UTC
Duration
—
Detected by Pingoru
Aug 24, 2023, 04:00 PM UTC

Update timeline

  1. resolved Aug 24, 2023, 07:59 PM UTC

    There were elevated response times in the US region for a small percentage of the requests. The issue was due to a memory leak in a new third party library we integrated with, which caused some nodes to operate with constrained resources (heavy CPU/MEM usage). This fixed itself automatically by killing the affected nodes (each one as it exhausted its resources over the span of a few minutes) and starting new ones instead. The team has also deployed a fix to revert the inclusion of the new library pending further investigation on why this was happening, and why it wasn't detected in the staging environment.