Timekit incident

Database server upgrade

Major Resolved View vendor source →

Timekit experienced a major incident on July 6, 2016, lasting 19h 4m. The incident has been resolved; the full update timeline is below.

Started
Jul 06, 2016, 01:46 PM UTC
Resolved
Jul 07, 2016, 08:51 AM UTC
Duration
19h 4m
Detected by Pingoru
Jul 06, 2016, 01:46 PM UTC

Update timeline

  1. scheduled Jul 06, 2016, 01:46 PM UTC

    We're upgrading our storage infrastructure and you may encounter some downtime between 2am to 3am PDT (9am and 10am UTC). We expect the maintenance period to be no more than 30 minutes, but most likely less.

  2. scheduled Jul 07, 2016, 06:59 AM UTC

    Maintenance will begin as scheduled in 60 minutes.

  3. in progress Jul 07, 2016, 08:00 AM UTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  4. in progress Jul 07, 2016, 08:28 AM UTC

    The machine snapshot is taking a bit longer that expected. The API is fully operational during this time, but it will postpone the actual machine upgrade about 30 minutes. New ETA is 9:30-10:00am UTC.

  5. monitoring Jul 07, 2016, 08:47 AM UTC

    Database upgrade completed, verifying that all systems are back to normal 👌

  6. completed Jul 07, 2016, 08:51 AM UTC

    Everything back to normal, now with double memory capacity! 🤘