Currencycloud experienced a minor incident on June 17, 2025 affecting Notifications, lasting 5h 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 17, 2025, 04:28 PM UTC
We're currently experiencing an issue where webhook notifications are not being delivered as expected. Our engineering team is actively investigating the root cause and working to resolve it as quickly as possible. We understand the importance of timely notifications and apologise for any inconvenience this may cause. We'll keep you updated with progress and notify you as soon as normal service is restored. Thank you for your patience and understanding.
- resolved Jun 17, 2025, 09:57 PM UTC
This incident has been resolved.
- postmortem Jun 30, 2025, 08:19 AM UTC
# Incident Summary On 17 June during a routine system upgrade, we encountered unexpected issues that disrupted how messages were processed and delivered across some of our services. # Root Cause * A Debian OS upgrade introduced compatibility issues affecting message handling * Our notifications system slowed down due to an inefficient database query * Some services had trouble maintaining stable connections to our messaging system \(RabbitMQ\) * Automatic retries overwhelmed the system, eventually redirecting messages into a backup queue # Remediation * Identified and fixed the database query causing the bottleneck * Adjusted messaging configurations to stabilize service connections * Increased system resources and improved retry logic to reduce congestion * Cleared affected queues and ensured all valid messages were processed Service restored same day, with impacted messages reissued between 18th and 20th June. Thank you for your patience as we worked to address the issue. If you're still experiencing any issues with missing messages, please don't hesitate to contact our support team directly.