FastSpring incident

Rebill Issues

Minor Resolved View vendor source →

FastSpring experienced a minor incident on April 19, 2022 affecting FastSpring System, lasting 6h 16m. The incident has been resolved; the full update timeline is below.

Started
Apr 19, 2022, 03:42 PM UTC
Resolved
Apr 19, 2022, 09:58 PM UTC
Duration
6h 16m
Detected by Pingoru
Apr 19, 2022, 03:42 PM UTC

Affected components

FastSpring System

Update timeline

  1. investigating Apr 19, 2022, 03:42 PM UTC

    We are currently investigating additional issues with some subset of rebills going back to Apr-18.

  2. monitoring Apr 19, 2022, 07:08 PM UTC

    Missing rebills from Apr-18 and Apr-19 are currently being rerun. The root cause is under investigation, but we have made changes to our architecture intended to avoid ongoing problems with our next rebill batch. Some subscriptions where a rebill was attempted on Apr-18 have entered an unusual state. We are working on a remediation plan to return the subscriptions to good standing.

  3. resolved Apr 19, 2022, 09:58 PM UTC

    All active rebills from 4/18 and 4/19 have been completed. Any rebill failures from 4/18 or 4/19 will be retried normally on the next rebill run on 4/20. We continue to monitor the system to ensure timely rebill runs. The root cause remains under investigation.