Currencycloud experienced a minor incident on June 18, 2025 affecting Notifications, lasting 2d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 18, 2025, 07:50 AM UTC
Due to the incident on the 17th June, we are investigating some clients missing important webhook notifications, after incident was mitigated. The team is working to identify the missing webhooks so they can be resent
- investigating Jun 18, 2025, 10:13 AM UTC
We are working to identify missing webhooks so they can be resent, this work is ongoing
- identified Jun 18, 2025, 01:48 PM UTC
We are in the process of resending the missing notifications; however, this may result in some clients receiving duplicate webhooks and emails. We apologies for any inconvenience this may have caused.
- monitoring Jun 18, 2025, 05:59 PM UTC
Webhooks have been resent, incident will remain open until all checks have been completed.
- monitoring Jun 19, 2025, 09:18 AM UTC
We are continuing to work on missing webhooks for clients All Funding webhooks have been resent and will not be sent again.
- resolved Jun 20, 2025, 05:17 PM UTC
The incident has been resolved. All missing webhooks have been resent successfully.
- postmortem Jun 30, 2025, 08:22 AM UTC
# Incident Summary On 17 June during a routine system upgrade, we encountered unexpected issues that disrupted how messages were processed and delivered across some of our services. # Root Cause * A Debian OS upgrade introduced compatibility issues affecting message handling * Our notifications system slowed down due to an inefficient database query * Some services had trouble maintaining stable connections to our messaging system \(RabbitMQ\) * Automatic retries overwhelmed the system, eventually redirecting messages into a backup queue # Remediation * Identified and fixed the database query causing the bottleneck * Adjusted messaging configurations to stabilize service connections * Increased system resources and improved retry logic to reduce congestion * Cleared affected queues and ensured all valid messages were processed Service restored same day, with impacted messages reissued between 18th and 20th June. Thank you for your patience as we worked to address the issue. If you're still experiencing any issues with missing messages, please don't hesitate to contact our support team directly.