Increased errors with Core Transactional API

Update timeline

1. investigating Mar 16, 2026, 05:45 PM UTC

   We are currently investigating elevated errors on API endpoints

2. monitoring Mar 16, 2026, 05:51 PM UTC

   A fix has been implemented and we are monitoring to ensure services are operating correctly

3. resolved Mar 16, 2026, 06:07 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. postmortem Mar 19, 2026, 07:07 PM UTC

   ### Mar 18th 2026 — _Decryption Errors on API endpoints_ ### Summary Between 5:40 PM and 5:43 PM UTC on March 16, 2026, we experienced a brief issue during a key rotation that caused a small subset of API requests to fail due to decryption errors. These errors occurred within our system before any external processing took place. The issue was quickly identified, and traffic was immediately redirected to a stable environment, restoring normal service. ### What Happened At 5:36 PM UTC, we routed traffic to a newly deployed cluster of our encryption service running with the latest security keys. Shortly after, we observed decryption errors affecting a small subset of records, which resulted in some requests returning 500 error responses. We promptly redirected traffic back to the existing cluster, restoring normal operation. Further investigation determined that these records were still associated with a key in the process of rotation, and we updated them as a follow-up action to prevent the issue from recurring. ### Next Steps We have enhanced our process to ensure that a key’s usage is fully verified before it is removed from the encryption service. -The Spreedly Team