Spreedly experienced a critical incident on September 30, 2025 affecting Core Transactional API, lasting 1h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 30, 2025, 06:22 PM UTC
We are currently investigating a high number of 500 errors
- investigating Sep 30, 2025, 06:25 PM UTC
Currently we are experiencing issues with services that use payment methods
- identified Sep 30, 2025, 06:55 PM UTC
We are implementing a mitigation that should restore the ability to perform new tokenizations and operations on those payment methods. Some functionality, such as fingerprinting, may remain limited. Use of vaulted payment methods may remain limited. We’ll provide updates as we progress.
- monitoring Sep 30, 2025, 07:07 PM UTC
We have identified and resolved the issue and and monitoring for any additional impact.
- resolved Sep 30, 2025, 07:45 PM UTC
The incident impacting operations involving payment methods has been resolved. All services are now operating normally. During routine database maintenance, we encountered issues that affected tokenization and transactions using payment methods. We’ll share more details in the upcoming post-mortem.
- postmortem Oct 02, 2025, 09:43 PM UTC
### Postmortem: _September 30, 2025_ — _Payment Method Services Inaccessible_ During a routine database configuration change, we lost connection to the Payment Methods storage table. This prevented the use of payment methods in transactions and the creation of new payment methods. ### What Happened During a routine database change to decommission extra nodes, a corruption occurred with our internal reference data. This corruption caused the database query optimizer to error when trying to process SQL queries on the payment methods storage, which resulted in the inability to access that storage table and led to errors from the application layer when trying to read, update, or create payment methods. ### Next Steps Payment methods successfully created during this period have had their fingerprints regenerated. Please contact Spreedly Support if you need the updated fingerprint values to utilize within your business processes and we will be happy to provide them. Spreedly is also working with our database service provider on a more detailed root cause analysis. Moving forward, Spreedly will be proactively engaging with our database provider when performing these types of configuration changes. -The Spreedly Team