Subscribe Pro incident

Background job processing (including ordering jobs) down

Minor Resolved View vendor source →

Subscribe Pro experienced a minor incident on October 9, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
Oct 09, 2022, 11:08 PM UTC
Resolved
Oct 04, 2022, 11:05 PM UTC
Duration
—
Detected by Pingoru
Oct 09, 2022, 11:08 PM UTC

Update timeline

  1. resolved Oct 09, 2022, 11:08 PM UTC

    Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause: * A key scheduling service failed. * This was triggered by our core transactional database recycling after a minor version upgrade. * This happened around 3am eastern time on 10/4 in us-east-1 region. * Happened around 3am eastern time on 10/1 in eu-west-1 region. * We should have noticed this, however there was an error in some of our alerting / notification scripts. * We have since updated and thoroughly tested the notification scripts.