Spreedly experienced a minor incident on May 14, 2025 affecting Alternative Payment Methods (APMs), lasting 13m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 14, 2025, 04:07 PM UTC
We are currently investigating some degradation with Google Pay transactions.
- monitoring May 14, 2025, 04:16 PM UTC
AFFECTED SYSTEMS OR SERVICES: Google Pay Transactions DETAILS OF THE INCIDENT: Degradation with Google Pay transactions CURRENT ACTIONS: We’ve implemented a fix and are seeing Google Pay transactions succeed. Our team is monitoring the situation closely to ensure there are no additional impacts.
- resolved May 14, 2025, 04:21 PM UTC
IMPACT STARTED AT: 11:18 AM EST IMPACT ENDED AT: 12:09 PM EST AFFECTED SYSTEMS OR SERVICES: Google Pay Transactions DETAILS OF THE INCIDENT: Google Pay Transactions Failing CURRENT ACTIONS: After closely monitoring Google Pay transactions and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.
- postmortem May 15, 2025, 10:00 PM UTC
# May 14, 2025 — HTTP 500 Errors and Google Pay Degradation ### High-Level Summary On May 14, 2025 at 03:18 pm UTC, a configuration change was deployed to enhance application fault tolerance which, resulted in a brief period of HTTP 500 errors. While the breaking change was reverted within a minute of deployment, there were still instances of cached data which extended the impact for a total of 4 minutes of partial service outage, until 03:22 pm UTC. A secondary issue occurred as a result of the timing of this deployment coinciding with the regeneration of the Google Pay verification keys for decryption, which resulted in a 51 minutes degradation of Google Pay transactions until 04:09 pm UTC. ### What Happened A configuration change intended to enhance application fault tolerance to Redis failures inadvertently introduced serialization issues when accessing Redis keys. This resulted in new cache failing to be successfully read from the existing application instances, which resulted in HTTP 500 errors. Additionally, at the same time this change was deployed, the Google Pay verification keys were regenerated. Google Pay decryption stopped working due to the cached keys not having serialized properly. These keys then had to be deleted from the cache to enable the system to auto generate a new pair. ### Next Steps * Spreedly has implemented additional observability improvements via our automated alerting system to enhance monitoring of Google Pay services.