BCH Digital experienced a major incident on July 30, 2025 affecting 💳 Payments, lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 30, 2025, 11:19 AM UTC
We are currently investigating error reports on the PCI platform.
- identified Jul 30, 2025, 11:26 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 30, 2025, 12:44 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 31, 2025, 12:23 PM UTC
On 30th July 2025, a misconfiguration in our cloud infrastructure led to a temporary service interruption that impacted a number of production systems. Now that 24 hours has passed without incident we can conclude our investigation. What Happened: During routine work as part of our PCI Compliance program, we identified unusual network behaviour that required investigation. As part of the resolution process, a planned change to internal network routing was prepared for staging and QA review. Unfortunately, due to human error, this change was inadvertently applied directly to our live production environment. This resulted in disrupted network connectivity across several core components, which caused a partial outage affecting web services, IVR, and transactional systems. Resolution: The change was identified and promptly reverted. Systems began recovering immediately and services were partially restored after 18 minutes (IVR, portal and automated txs) fully restored after 51 minutes (VT live agent). What We’re Doing to Prevent It: We’ve introduced additional safeguards to prevent this from happening again, including: - Improved environment tagging and stricter validation checks before any infrastructure changes are made - Enhanced peer-review requirements for all production-related deployments - Internal procedural updates around deployment and QA staging We sincerely apologise for the disruption and appreciate your understanding while we worked to resolve the issue.