Svix incident

Subtle changes in how payloads are sent broke signature verification for consumers verifying incorrectly

Notice Resolved View vendor source →

Svix experienced a notice incident on April 10, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Apr 10, 2023, 04:28 PM UTC
Resolved
Apr 09, 2023, 11:00 AM UTC
Duration
Detected by Pingoru
Apr 10, 2023, 04:28 PM UTC

Update timeline

  1. resolved Apr 10, 2023, 04:28 PM UTC

    We changed the code to send the payload exactly the same way as it's sent to us (before, we were compacting it before sending). This meant that people that relied on the payload to be compact in order to verify webhooks (i.e they were verifying incorrectly) had verification failing. We reverted this immediately once it was reported signatures were failing for customers. While not a bug in Svix, as people verifying webhooks correctly wouldn't have had issues, it still caused disruption to our customers and we've added tests to ensure that we now always compact the payloads going forward. We are also working on making it even harder to get wrong for customers, but please refer to https://docs.svix.com/receiving/verifying-payloads/how for the correct way of verifying webhooks.