S2 - Eptura Workplace - Sites are experiencing an error while navigating through the site.

Update timeline

1. investigating Sep 17, 2025, 01:50 PM UTC

   We are currently investigating an issue with sites that are experiencing errors while navigating through the site.. Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 11 AM CST

2. identified Sep 17, 2025, 02:00 PM UTC

   We are moving into an identified phase. Our teams have identified that the issue is related to a release deployed last night. To mitigate the errors and restore stability, we are actively rolling back the release. We are now moving into the identified phase of our incident response and are working with urgency to resolve the issue. We understand how critical this is to your operations and appreciate your patience as we work through it. Further updates will be shared as progress continues.

3. monitoring Sep 17, 2025, 02:47 PM UTC

   We’ve completed the rollback of the release deployed last night, and all production servers have been successfully restored. At this time, your operations should be back to normal. We truly appreciate your patience and understanding during this disruption. Our teams will continue to monitor the environment closely over the next four hours to ensure full stability and performance. Thank you for your continued trust.

4. resolved Sep 17, 2025, 06:50 PM UTC

   We have moved to the Resolved Phase. As we have not seen further service disruptions after the rollback. Thank you for your patience and understanding while we worked to address the issue. We sincerely apologize for any inconvenience caused.

5. postmortem Sep 29, 2025, 10:19 PM UTC

   **Eptura Workplace detailed Root Cause Analysis | 9/17/2025** **S2 - End users are experiencing an error while navigating through Eptura Workplace** We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** Multiple customers experienced a temporary disruption in Eptura Workplace due to a release that was deployed to production on the evening of 09/16/2025 at approximately 10 PM CT. **Type of Event:** Intermittent Outage **Services/Modules impacted:** Eptura Workplace Production Environment **Timeline in CT time zone:** ` ` 2:46 AM – An initial support case was reported regarding difficulty accessing the platform. The customer was able to log into their account but could not access the reservations module. Attempts to refresh the page resulted in continuous loading without resolution. 6:30 AM – Two additional support cases were submitted, this time indicating complete site inaccessibility. Support escalated the issue by creating a Severity 2 JIRA ticket with Cloud Operations to begin investigation \(CM-98483\). 8:30 AM – Internal reports surfaced indicating that the AUP was down and the internal Server for the US region was experiencing issues. A new JIRA \(CM-98491\) was created, and Cloud Ops confirmed the issue was related to CM-98483. Internal discussions pointed to a recent release for Eptura Workplace deployed on the evening of 9/16 as the likely cause. A rollback was considered as a potential resolution. 8:50 AM – A fire alarm was triggered, and all customers were notified of the disruption via the Eptura Workplace Status Page. The message indicated that the issue was under active investigation. 9:00 AM – Customers were updated that the root cause had been identified. To resolve the issue, a rollback of the 9/16 release was required. 9:47 AM – The rollback was successfully completed. The system entered a monitoring phase, during which QA performed a sanity check. The check was completed successfully, confirming system stability. 1:50 PM – The incident was officially moved to the Resolved Phase. No further service disruptions were observed following the rollback. **Total Duration of Event:** 11 hrs 4 mins **Root Cause:** During a recent release, we introduced a code change that required specific data fields from customer feeds. These fields were present in our internal test environments, and as a result, the new code passed all internal testing successfully. However, upon deployment to production, we discovered that some customer feeds did not include these required fields. This discrepancy led to a data missing exception in the new code. Unfortunately, this exception triggered a pre-existing issue in the system, where database connections were not being properly released. As more exceptions occurred, these unreleased connections accumulated over time, eventually exhausting the available database connections and causing a temporary system outage. **Remediation:** To address the issue, we took the following steps: 1. **Resolved the underlying bug**: We fixed the bug that was causing database connections to remain open without being properly released. 2. **Deployed a patch**: We implemented a patch to ensure that the new code handles missing data fields more gracefully, preventing similar exceptions from cascading into system-level failures. 3. **Restarted affected services**: We restarted the affected services and cleared stale connections to restore system stability. These immediate actions helped mitigate the impact and restore normal operations. **Preventative Action:** To prevent this issue from recurring, we are implementing several key improvements: 1. **Enhanced validation processes**: We are improving our validation processes to ensure that all required data fields are present and accounted for before any new code is deployed. 2. **Expanded test coverage**: We are broadening our test coverage to better simulate production environments, including the variability in customer data feeds. 3. **Monitoring and alerting**: We are introducing new monitoring and alerting mechanisms to detect abnormal database connection usage early, allowing us to take proactive measures. 4. **Safeguards for connection leaks**: We are adding additional safeguards to detect and recover from connection leaks more effectively, ensuring greater resilience in future releases. These improvements are designed to enhance the reliability and stability of our system, and we are committed to delivering a seamless experience for all our customers.