S2 - Visitor - Inability to use the Checkout and Delete functionality from the Queue.

Update timeline

1. investigating Jul 12, 2023, 04:11 PM UTC

   We are currently investigating this issue.

2. investigating Jul 12, 2023, 06:20 PM UTC

   We continue to investigate an issue with the Visitor Module and the Inability to use the Checkout and Delete functionality from the Queue.. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 4:20pm EDT.

3. investigating Jul 12, 2023, 08:32 PM UTC

   We continue to investigate an issue with the Visitor Module and the Inability to use the Checkout and Delete functionality from the Queue.. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 6:30pm EDT.

4. investigating Jul 12, 2023, 10:55 PM UTC

   We are continuing to investigate this issue. Our next update will be posted 8:55pm EDT.

5. investigating Jul 13, 2023, 01:06 AM UTC

   Investigation will continue on the issue with our Visitor Module and the Inability to use the Checkout and Delete functionality through this evening. Additional updates will continue tomorrow, Thursday, 07/13/2023, 8:00am EDT.

6. investigating Jul 13, 2023, 01:41 PM UTC

   Our Engineering team is continuing to investigate to determine the cause of the disruption. The next update will be posted at 11:00 AM EST

7. investigating Jul 13, 2023, 03:32 PM UTC

   Investigation continues on for this issue. Our next update will be provided at approximately 1:30pm EDT. We appreciate your patience as we work through this disruption.

8. investigating Jul 13, 2023, 05:43 PM UTC

   Our teams continue to put much effort into the investigating the root cause. We will continue to update you at approximately, 3:30 EDT. Thank you.

9. identified Jul 13, 2023, 07:48 PM UTC

   The issue with Visitor - Inability to use the Checkout and Delete functionality from the Queue has been identified and our internal teams are working on the fix. We will post another update at approximately 6pm EDT.

10. identified Jul 13, 2023, 10:16 PM UTC

    Our internal teams continue to work on the resolution for Visitor - Inability to use the Checkout and Delete functionality from the Queue. Our next updates will be provided at approximately 8pm EDT.

11. identified Jul 14, 2023, 12:19 AM UTC

    We are continuing to work on a fix through the remainder of this evening and will provide more updates tomorrow morning, Friday, July 14 at approximately 9am EDT. Thank you.

12. monitoring Jul 14, 2023, 11:56 AM UTC

    A fix has been implemented and we are monitoring the results.

13. resolved Jul 14, 2023, 02:40 PM UTC

    This incident has been resolved.

14. postmortem Jul 26, 2023, 07:47 PM UTC

    **iOFFICE Detailed Root Cause Analysis – Severity 2 – July 12, 2023** **Description:** On July 12, 2023, at approximately 12:11pm EDT, internal teams and customers support started to receive excessive reports of issues throughout the iOffice platform with the inability to use the Checkout and Delete functionality in the Visitor Module. End users would experience errors when clicking on the checkout function in the Visitor queue. **Type of Event:** Service Disruption **Services\\Modules Impacted:** Production – Visitor Module **Timeline:** On Wednesday, July 12, 2023, at approximately 12:11pm EDT, internal teams and customers support started to receive excessive reports of issues throughout the iOffice platform with the inability to use the Checkout and Delete functionality in the Visitor Module. End users would experience errors when clicking on the checkout function in the Visitor queue. All customers were alerted via Status Page about the disruption. Investigation continued through Thursday, July 13, 2023, till 6:13pm EDT when engineers had identified the issue. Our engineering team had Implemented a fix at approximately 9:10pm and monitored the system for the next hour. Friday, July 14, 2023, at 7:56am EDT support began reaching out to customers for confirmation. At 10:40am EDT the Status Page was moved from Monitoring to Resolved. ‌ **Total Duration of Event:** 1 Day, 22 Hours, 29 Minutes **Remediation:** Change critical variable to be re-calculated at every use, instead of on class instantiation. This resolves the malformed database connection and records are found as needed. **Root Cause Analysis:** Internal teams found the root cause was introduced while resolving 2 iOffice issues. The modified Visitor Module logic to set a critical variable needed by the visitor service at the class level. In production usage volume, it was found that this class is at times reused instead of created per request as expected, which created a malformed database connection within the visitor service. The service then could not find the record to modify and would return an error. **Preventative Action:** No further steps are needed to take preventative action.