Spreedly incident
Early Alert: Potential Customer-Impacting Issue Detected
Spreedly experienced a major incident on March 15, 2025 affecting Core Transactional API, lasting 6h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 15, 2025, 06:50 AM UTC
We are investigating an elevated rate of 5xx errors affecting Spreedly core services starting at 01:36AM EST. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Our team is actively working to identify the root cause and mitigate the impact. We will provide further updates as soon as we have more information. Thank you for your patience.
- monitoring Mar 15, 2025, 07:32 AM UTC
As of 3:10 AM EST, our system availability has been fully restored and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.
- monitoring Mar 15, 2025, 11:35 AM UTC
As of 3:10 AM EST, our system availability has been fully restored and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.
- resolved Mar 15, 2025, 01:40 PM UTC
IMPACT STARTED AT: 2:12 AM EST IMPACT ENDED AT: 2:40 AM EST After closely monitoring Spreedly Core Services and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.
- postmortem Mar 21, 2025, 02:07 AM UTC
# March 15th, 2025 — Payment Method Storage Errors _On March 15th, 2025 at 06:12 UTC Spreedly systems encountered a system resource constraint which led to intermittent API call failures resulting in_ `404` _and_ `500` errors being returned to customers. _The unresponsive application containers were restarted allowing the system to return to normal at 06:47 UTC_ ## What Happened On 2025-03-15 between 06:12 and 06:47 UTC, calls to Spreedly’s payment method tokenization endpoints intermittently failed resulting in `HTTP 500` errors. Intermittent elevated rates of `HTTP 404` errors when attempting to use a payment method to transact were also encountered. These failures were the result of internal system resource limitations. ## Next Steps * Spreedly has implemented additional resource monitoring in our automated alerting system to prevent future overruns. * System resources have been adjusted to prevent similar situations in the future.