Spreedly experienced a minor incident on December 10, 2024 affecting iFrame, lasting 3h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 10, 2024, 04:59 PM UTC
Spreedly has discovered an issue that could potentially impact our services, leading to errors with caching the CVV. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We provide an update as soon as more details are available. Thank you for your patience.
- monitoring Dec 10, 2024, 05:16 PM UTC
Spreedly experienced an issue with Caching the CVV. Once identified, we immediately began working on a resolution. We’ve implemented a fix and stabilized the system. Our team is monitoring the situation closely to ensure there are no additional impacts. Please reach out to our Support team if you need any additional help.
- resolved Dec 10, 2024, 08:22 PM UTC
After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. A post-mortem review will be published. We apologize for any inconvenience or disruption.
- postmortem Dec 12, 2024, 10:32 PM UTC
## December 10th, 2024 — CVV Caching Issue At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from storing CVVs for payment methods when a /recache request was made. The adverse change resulted in a partial outage of Spreedly’s transactional API between 15:36 UTC and 17:07 UTC. Service was fully restored at 17:07 UTC. After a period of monitoring, Spreedly’s systems are stable, and no further impact is expected on customers. ## What Happened At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from extending the time a CVV exists in the cache. The adverse change resulted in a subset of transactions failing for customers who rely on Spreedly’s CVV recaching mechanism to complete a transaction. Spreedly Engineering team quickly mobilized and identified the adverse code deployment as the issue and immediately began reverting the change. The code reversion was completed at approximately 17:07 UTC, with customers reporting no additional impact after the code reversion. ## Next Steps Spreedly Engineering is reviewing internal observability to determine how to automate the detection of such an issue in the future.