S2 - Functionality issues within the iOffice application derived from the Product Release 07/06/22

Update timeline

1. investigating Jul 07, 2022, 04:25 PM UTC

   There have been some reports around functionality issues within the iOffice application derived from the Product Release of last night. Our Product and Engineering teams are working quickly to address and determine a path forward to resolution as soon as possible. You can find further updates on the iOffice Status Page moving forward. Please feel free to reach out to your CSM for further information.

2. identified Jul 07, 2022, 08:07 PM UTC

   We have completed all regression tests and will be starting a rollback of this release to be able to address theses issues and return functionality. Next update will be at 4:00pm CST

3. monitoring Jul 07, 2022, 08:48 PM UTC

   Rollback is completed and we are seeing the functionality previously experienced has returned, thank you for your understanding. We will continue to monitor this event and compete some additional testing before resolving, Next update 5pm CST

4. resolved Jul 07, 2022, 10:01 PM UTC

   We're now moving to a resolved state as our Support team is seeing improved performance and proper functionality with the product. We appreciate your continued patience while our Engineering team resolved this issue.

5. postmortem Jul 14, 2022, 06:18 PM UTC

   On July 7, 2022, at approximately 8:30am MST, internal teams and customers support started to receive reports of issues throughout the iOffice platform with toggle buttons and admin functions. **Root Cause & Remediation**: Internal teams were alerted to and identified that issues were derived from the Product Release on July 6, 2022. After attempts to remediate were unsuccessful it was decided that a release roll back was necessary and was completed around 2:46pm MST. **Timeline:** 8:30am MST – Internal teams and customers have reported multiple issues regarding the product release the day prior. After a short investigation and information gathering to reproduce customers experiences. The initial ticket was escalated to a S2 incident at 10:24am MST and posted to our status page alerting out customers. internal teams acknowledge the issue and begin regression testing against previous product build. 11:18am MST testing continues, and decision is made to roll back release, 1:30pm MST Roll back of product release begins and is completed at 2:46pm MST. Beginning a monitoring and testing phase of the incident, after verifying with multiple customers and confirming the roil back was successful, an all-clear was declared and incident closed at 4:02pm MST. **Preventative Action and Analysis**: Internal teams have discovered opportunities in their release process around tooling and QA that need improvement. They are also reviewing adherence to their internal policies that could have prevented some of the issues that were discovered.