Subscribe Pro incident

Background job processing (including ordering jobs) down

Resolved on October 4, 2022.

resolved Oct 09, 2022, 11:08 PM UTC

Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause: * A key scheduling service failed. * This was triggered by our core transactional database recycling after a minor version upgrade. * This happened around 3am eastern time on 10/4 in us-east-1 region. * Happened around 3am eastern time on 10/1 in eu-west-1 region. * We should have noticed this, however there was an error in some of our alerting / notification scripts. * We have since updated and thoroughly tested the notification scripts.

Started
Oct 04, 2022, 11:05 PM UTC
Resolved
Oct 04, 2022, 11:05 PM UTC
Duration
–
Detected by Pingoru
Oct 04, 2022, 11:05 PM UTC

Update timeline

  1. resolved Oct 09, 2022, 11:08 PM UTC

    Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause: * A key scheduling service failed. * This was triggered by our core transactional database recycling after a minor version upgrade. * This happened around 3am eastern time on 10/4 in us-east-1 region. * Happened around 3am eastern time on 10/1 in eu-west-1 region. * We should have noticed this, however there was an error in some of our alerting / notification scripts. * We have since updated and thoroughly tested the notification scripts.

Your product runs on more than Subscribe Pro

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →