Baremetrics experienced a notice incident on October 22, 2020 affecting Stripe API, lasting 1h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 22, 2020, 01:48 PM UTC
We've detected that roughly half of all webhooks from Stripe are getting blocked. This appears to be a new issue with the way CloudFlare handles rate-limiting. In this case, they're incorrectly limiting calls from Stripe's IP addresses. We're working with them now to resolve this. All failing webhooks will automatically be reprocessed once this is resolved. We'll provide more updates as we have them.
- identified Oct 22, 2020, 02:13 PM UTC
We've identified the issue is indeed with CloudFlare's rate-limiting incorrectly blocking IP addresses that we've whitelisted. Working with CloudFlare now to address this.
- identified Oct 22, 2020, 02:24 PM UTC
We've deployed a temporary fix that should allow Stripe API calls to process while we work with CloudFlare.
- monitoring Oct 22, 2020, 02:43 PM UTC
CloudFlare reports they've resolved the issue on their end. We're monitoring their fix now, but otherwise webhooks should be processing normally now and we're working our way through the backlog of previously failed webhooks.
- resolved Oct 22, 2020, 03:08 PM UTC
Everything seems to be back in order and we've confirmed all webhooks are properly processing now.