FastSpring incident
Tags incorrect in subscription.activated webhook
FastSpring experienced a minor incident on April 5, 2023 affecting FastSpring System, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 06, 2023, 12:33 PM UTC
Order-level tags are being passed incorrectly to the subscription.activated webhooks. It is important to note that order-level tags kept being passed correctly to the order.completed webhooks.
- monitoring Apr 06, 2023, 12:37 PM UTC
We have rolled back the code that caused this change in behavior. Our Engineering team is working on mitigating actions for the impacted susbcription.activated webhooks. If the order.completed webhook is enabled at store-level on your account, we recommend using it as source of truth for the order-level tags of subscriptions created during the incident timeframe.
- monitoring Apr 06, 2023, 03:48 PM UTC
We have fully identified and verified the root cause of the problem. We have also verified the data is not permanently lost. We are confident we will be able to regenerate the appropriate data and send the webhooks with the correct tags. We should have a cleanup plan in place with tools within 24 hours.
- resolved Apr 06, 2023, 05:00 PM UTC
Data cleanup is in process. Please contact support if you need subscription.activated webhooks regenerated from orders during this timeframe or if you have additional issues or concerns. We will not be refiring proactively to avoid causing you problems with duplicates, but we can do it on a case-by-case basis. Correct data can also be pulled via API call or we can supply a csv of the data.