Perk Box incident

Australia Platform maintenance - performing rollback

Critical Resolved View vendor source →

Perk Box experienced a critical incident on February 6, 2020 affecting Perks and Login and 1 more component, lasting 20h 18m. The incident has been resolved; the full update timeline is below.

Started
Feb 06, 2020, 06:09 PM UTC
Resolved
Feb 07, 2020, 02:28 PM UTC
Duration
20h 18m
Detected by Pingoru
Feb 06, 2020, 06:09 PM UTC

Affected components

PerksLoginCelebrationsAdminEmailsInfrastructure

Update timeline

  1. investigating Feb 06, 2020, 06:09 PM UTC

    Due to an issue found during the maintenance, we are now rolling back our changes and will then reschedule the maintenance work to be carried out at another time.

  2. identified Feb 06, 2020, 09:18 PM UTC

    We’re experiencing technical issues at the moment. We’re working hard to fix the problem - please try again later. Thank you for your patience

  3. identified Feb 07, 2020, 06:44 AM UTC

    Our engineers have been tirelessly working through the night to bring you back the service you know and love. We’re very very nearly there… please sit tight for further updates! This is not the service we strive to provide for our customers and thank you all for you patience and support.

  4. identified Feb 07, 2020, 08:43 AM UTC

    Good news - our Perks and Recognition products are now back up and running and you should be able to use these two products as usual! :tada: Great work tech wizards. We are still experiencing issues with our Insights product, but are working hard on a fix. We’ll be back on here to let you know when it’s back with us. Thank you all again for your patience - we really appreciate it!

  5. resolved Feb 07, 2020, 02:28 PM UTC

    Our Insights product is fixed too. We have been monitoring our systems for the past few hours and everything is now back to normal. Thank you for your patience. This incident report is now closed.