Power Home Remodeling incident

Production Nitro upgrade to Ruby 2.2

Resolved after 15 days 17 hours, on May 17, 2016.

completed Jun 02, 2016, 10:03 AM UTC

Deployment completed, Nitro is being served by Ruby 2.2.

Started
May 17, 2016, 04:04 PM UTC
Resolved
Jun 02, 2016, 10:03 AM UTC
Duration
15d 17h
Detected by Pingoru
May 17, 2016, 04:04 PM UTC

Update timeline

  1. scheduled May 17, 2016, 04:04 PM UTC

    Production Nitro servers will be upgraded to Ruby 2.2 (https://github.com/powerhome/puppet-ctrl/pull/252). This will include a short service interruption of approximately 10 minutes while the application is re-deployed on the new Ruby version.

  2. scheduled May 27, 2016, 05:28 PM UTC

    Tim confirms the time. Ready to go.

  3. scheduled May 31, 2016, 08:29 AM UTC

    Maintenance will begin as scheduled in 60 minutes.

  4. scheduled May 31, 2016, 09:15 AM UTC

    Pushed back by a day due to lack of monitoring availability.

  5. in progress Jun 01, 2016, 09:31 AM UTC

    Upgrade being rolled out.

  6. in progress Jun 01, 2016, 09:48 AM UTC

    New Ruby package not available in production repo; rectifying and re-attempting upgrade.

  7. scheduled Jun 01, 2016, 10:05 AM UTC

    Upgrade attempt aborted; Puppet was running too slow to complete the upgrade within the maintenance window. Re-scheduled for a second attempt tomorrow.

  8. in progress Jun 02, 2016, 09:30 AM UTC

    Upgrade being rolled out.

  9. in progress Jun 02, 2016, 09:39 AM UTC

    Ruby upgrade is deployed to all nodes. Nitro deployment is rolling: http://ci.powerhrg.com/job/NitroProductionDeployment/2516/

  10. completed Jun 02, 2016, 10:03 AM UTC

    Deployment completed, Nitro is being served by Ruby 2.2.

Your product runs on more than Power Home Remodeling

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →