Apideck incident
Delays in webhook event processing with connectors that utilize Virtual Webhooks
Apideck experienced a minor incident on March 4, 2025 affecting Webhooks, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 04, 2025, 07:44 AM UTC
We are currently investigating this issue.
- resolved Mar 05, 2025, 08:23 AM UTC
This incident has been resolved.
- postmortem Mar 06, 2025, 08:26 PM UTC
Last week, we identified an issue in our system that led to delays in webhook event processing and manual sync triggers. A backlog of messages accumulated in our processing queue, causing high resource utilization and degraded performance. **Root Cause** The issue was traced to a bug in our pagination logic when processing data. This caused an unexpected loop, leading to repeated processing attempts and message buildup in our queue. **Resolution** We implemented a series of fixes, including: * Correcting the pagination logic to prevent looping * Optimizing message processing and error handling * Enhancing system monitoring to detect similar issues earlier **Impact** During the incident, some customers experienced delays in receiving webhook events and manually triggering syncs. We have since resolved the issue and taken steps to prevent recurrence. **Next Steps** We are implementing additional monitoring and safeguards to improve system resilience and ensure seamless webhook processing. We appreciate your patience and remain committed to maintaining a reliable experience.