Power Home Remodeling incident

Changes to deployment of Nitro cron jobs

Resolved after 1 day 18 hours, on May 11, 2016.

completed May 13, 2016, 10:05 AM UTC

Jobs are running.

Started
May 11, 2016, 03:54 PM UTC
Resolved
May 13, 2016, 10:05 AM UTC
Duration
1d 18h
Detected by Pingoru
May 11, 2016, 03:54 PM UTC

Update timeline

  1. scheduled May 11, 2016, 03:54 PM UTC

    According to https://www.pivotaltracker.com/story/show/116998393 and https://github.com/powerhome/nitro-web/pull/2021, the way in which Nitro deploys cron jobs will change. No downtime is expected, but there is the potential for mistakes to have been made and some jobs to not run correctly. Please report all suspicion that scheduled tasks are not running correctly immediately.

  2. scheduled May 12, 2016, 08:29 AM UTC

    Maintenance will begin as scheduled in 60 minutes.

  3. in progress May 12, 2016, 10:30 AM UTC

    A late start means we're going to have to carefully run some missed tasks manually, but we're rolling now.

  4. scheduled May 12, 2016, 10:34 AM UTC

    Aborted because broken code was committed to Nitro master last night. Re-scheduled for tomorrow.

  5. scheduled May 13, 2016, 08:36 AM UTC

    Production deployments are being suspended in preparation for this rollout.

  6. scheduled May 13, 2016, 08:56 AM UTC

    Change has been merged to Nitro master and is going through testing in CI. QA jobs are being removed in order to prepare for automatic QA deployment on completion.

  7. in progress May 13, 2016, 08:57 AM UTC

    Cron jobs are being removed in all demo/DevTB environments. Please do not manually deploy crontabs there; instead rebase your branches on master and let Whenever do it.

  8. in progress May 13, 2016, 09:06 AM UTC

    Dev1's crontab was also removed.

  9. in progress May 13, 2016, 09:20 AM UTC

    QA crontab is clear for deployment; no more jobs listed until 6am. Waiting on CI to deploy. Once a crontab is successfully installed in QA, production's crontab will be removed and a production deployment initiated.

  10. in progress May 13, 2016, 09:39 AM UTC

    QA deployment started: http://ci.powerhrg.com/job/Nitro-QA-Deployment/1997/

  11. in progress May 13, 2016, 09:44 AM UTC

    A crontab was successfully placed in QA. Production crontab removed and deployment started.

  12. in progress May 13, 2016, 09:48 AM UTC

    Production crontab was placed; awaiting first job at 6am (Project.set_add_on_potential!).

  13. monitoring May 13, 2016, 09:49 AM UTC

    Verification is currently underway for the maintenance items.

  14. completed May 13, 2016, 10:05 AM UTC

    Jobs are running.

Your product runs on more than Power Home Remodeling

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →