ACME Technologies experienced a major incident on March 13, 2025, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Mar 17, 2025, 09:18 PM UTC
ACME experienced an interruption that affected our ability to connect to endpoints required to properly complete Fraud Shield transactions. This was due to a rotation of Root CA and Leaf certificates by the solutions provider that occurred in an non-orthodox manner that caused ACME's systems to reject the new endpoints. Leadup and Resolution: Wednesday, March 12th incident 2025-03-12 15:52 UTC (11:52 AM Eastern) first notification to DevOps of a SSLHandshakeException 2025-03-12 15:52 UTC (11:52 AM Eastern) SSLHandshakeException errors continue for approximately an hour; the problem is that Cardinal changed their certs. 2025-03-12 17:00 UTC (14:00 PM Eastern) Cardinal rolls back their certs and 3DS at ACME stops throwing SSLHandshakeExceptions. Thursday, March 13th incident 2025-03-13 14:20 UTC (10:20 AM Eastern) first notification to DevOps that the SSLHandshakeException is again happening. Over the next few hours, ACME sees certificate upgrades and rollbacks on Cardinal endpoints, causing intermittent 3DS success/failure. 2025-03-13 19:14 UTC (15:14 Eastern) ACME update: ACME turns off 3DS for all venues; Cardinal also says they’ve rolled back the changes that caused problems. 2025-03-13 20:11 UTC (16:11 Eastern) DevOps sees SSLHandshakeExceptions return 2025-03-13 20:52 UTC (14:52 Eastern) ACME updates leaf cert in prod 2025-03-13 21:29 UTC (17:29 Eastern) ACME successful test in prod 107 2025-03-13 22:00 UTC (18:00 Eastern) ACME prod returns to calm 2025-03-13 22:00 UTC (18:00 Eastern) ACME production 3DS re-enabled for B2C whitelabel
- postmortem Mar 17, 2025, 09:18 PM UTC
Postmortem notes: ACME is examining a way to perform a graceful degradation component in Fraud Shield B2C and SDK where affected checkout flows can be maintained, with diminished liability requirements. Root Certificate workflow is being examined to implement additional automation as needed.