Clearbit incident

Webhook processing delayed

Major Resolved View vendor source →

Clearbit experienced a major incident on May 3, 2025 affecting Webhooks, lasting 2d 17h. The incident has been resolved; the full update timeline is below.

Started
May 03, 2025, 12:35 AM UTC
Resolved
May 05, 2025, 05:36 PM UTC
Duration
2d 17h
Detected by Pingoru
May 03, 2025, 12:35 AM UTC

Affected components

Webhooks

Update timeline

  1. investigating May 03, 2025, 12:35 AM UTC

    We're currently investigating a potential issue with webhook processing. Webhooks may not be delivering as expected. This issue appears to have started around 8:00 PM UTC (1:00 PM Pacific Time). We'll provide updates as we learn more.

  2. monitoring May 03, 2025, 01:03 AM UTC

    We've identified and resolved the issue affecting webhook processing. Queued webhooks are now being delivered, and the system is catching up. We'll continue to monitor performance closely until processing is fully up to date.

  3. monitoring May 03, 2025, 07:14 PM UTC

    While monitoring the issue, we discovered that the earlier fix did not fully resolve the problem. As a result, webhooks submitted between 1:00 AM and 7:00 AM UTC (6:00 PM to 12:00 AM Pacific Time) were not processed and cannot be recovered. To ensure delivery, these webhooks will need to be re-submitted. We implemented a fix, and are again monitoring.

  4. resolved May 05, 2025, 05:36 PM UTC

    This incident has been resolved.