S2 - Serraview - Intermittent Delays Navigating Serraview

Update timeline

1. investigating Mar 23, 2026, 07:10 PM UTC

   We are currently investigating an issue with intermittent errors navigating Serraview, primarily in the reporting module in both Production and UAT environments. Our Engineering team is actively working to determine the root cause of the disruption and assess its impact. We will provide our next update by 11:10 PM UTC (6:10 PM CDT). Thank you for your patience as we work to resolve this issue.

2. investigating Mar 23, 2026, 11:31 PM UTC

   Our engineering teams continue to investigate the disruption with intermittent errors navigating Serraview, primarily in the reporting module in both Production and UAT environments. We appreciate your patience as we work to identify the root cause. We will provide our next update by 03:31 AM UTC (10:31 PM CDT).

3. investigating Mar 24, 2026, 05:12 AM UTC

   We are continuing to investigate intermittent issues affecting navigation and data loading within Serraview, particularly across the Reporting and Spaces modules. Users may experience delays, unresponsive pages, or errors that are temporarily resolved upon refreshing the browser. Our Engineering team is actively working to identify the underlying cause and assess the full scope of impact across environments. We will share further updates as soon as more definitive information becomes available. Thank you for your continued patience.

4. identified Mar 24, 2026, 08:55 AM UTC

   We have identified the root cause of the issue impacting navigation and data loading within Serraview. The behaviour is related to session handling, which may result in users experiencing errors or being required to refresh or re-login during usage. Our Engineering team is actively prioritising a hotfix to address this and restore a seamless user experience. In the meantime, a browser refresh or re-login may help restore normal functionality. We will continue to share updates as we make progress, with the hotfix being treated as a priority. Thank you for your continued patience.

5. identified Mar 24, 2026, 02:08 PM UTC

   A hotfix has been developed to address this issue and is currently undergoing testing. Once validation is complete, the fix will be deployed to the Production environment. We will share further updates as more definitive information becomes available. Thank you for your continued patience.

6. identified Mar 25, 2026, 09:29 AM UTC

   A hotfix has been successfully validated in the UAT environment and we are now proceeding with deployment to the Production environment. We will share a further update once the deployment is complete. Following this, we will reach out on the respective cases to confirm functionality. Thank you for your continued patience while we work towards an expedited resolution.

7. monitoring Mar 25, 2026, 04:49 PM UTC

   A hotfix has been successfully deployed to the Production environment and has passed initial validation. With the deployment now complete, we have transitioned this incident into a monitoring phase to ensure continued stability and expected functionality. Our team will be reviewing the environment closely and will follow up on the respective cases to confirm that all impacted functionality has been fully restored. Thank you for your continued patience while we worked toward an expedited resolution.

8. monitoring Mar 25, 2026, 11:14 PM UTC

   Our monitoring of the UAT and Production environment continues to show stable performance following the hotfix deployment. Internal and external checks continue to confirm that the affected functionality is operating as expected, and no new issues have been observed. Our team will proceed with additional validation throughout the evening and will continue reviewing related support cases to ensure all impacted customers are fully restored. We appreciate your patience and will provide another update if any unexpected behavior arises.

9. monitoring Mar 26, 2026, 08:50 AM UTC

   Post deployment of the hotfix, both UAT and Production environments continue to remain stable with no further issues observed. Validation checks, both internal and external, indicate that the impacted functionality is working as expected. We will continue to monitor the environments closely and review associated support cases to ensure full recovery for all impacted customers. Further updates will be shared if any anomalies are identified. Thank you for your continued patience.

10. investigating Mar 26, 2026, 05:08 PM UTC

    Following our post‑deployment monitoring, we have observed continued intermittent slowness and delays for some customers. As a result, we are moving this incident back into an Investigating phase while our engineering teams work to identify the underlying cause and drive a full resolution. We sincerely apologize for the ongoing disruption and appreciate your patience as we continue working to restore optimal performance. Additional updates will be shared here as they become available.

11. monitoring Mar 26, 2026, 09:04 PM UTC

    We have successfully redeployed the hotfix to all US customers, and impacted systems are now fully operational. We are now transitioning back into a monitoring phase and will continue to closely monitor system performance throughout the evening to ensure continued stability. Our Support team will also maintain proactive outreach on any related cases to ensure every customer is fully supported. We sincerely appreciate your patience and understanding as we worked through this disruption. Thank you for your partnership.

12. monitoring Mar 30, 2026, 09:57 AM UTC

    The hotfix has now been redeployed across all regions, and services are currently stable. We remain in the monitoring phase and continue to closely observe system performance to ensure consistency. Our teams are actively reviewing ongoing cases to confirm that the issue has been fully resolved. We request you to please validate from your end and confirm on your existing case. If you continue to experience any issues, please update the case or raise a new one, and we will prioritise it accordingly. We will continue to keep you informed as we progress. Thank you for your continued cooperation.

13. resolved Mar 31, 2026, 09:11 AM UTC

    We are pleased to confirm that the issue has now been resolved following the successful deployment of the hotfix across all regions and services are operating as expected. If you continue to experience any issues, please update your existing case or raise a new one and our team will prioritise it accordingly. We sincerely appreciate your patience and understanding while we worked to restore normal operations. A detailed Root Cause Analysis will be published within the next 10 business days. Thank you for your continued trust and partnership.

14. postmortem May 27, 2026, 09:02 AM UTC

    We appreciate your patience and continued support while we completed the investigation into the intermittent Serraview performance and search-related issues. **Type of Event:** Severity 2 – Intermittent delays navigating Serraview **Services/Modules Impacted:** Serraview Spaces, VBS, and Scenario Planner. **Issue Summary / Background:** Some customers experienced intermittent issues within Serraview, including unresponsive search functionality, loading errors, and delays when navigating across modules such as Spaces, VBS, Reporting, and Scenario Planner. The issue typically occurred after a period of inactivity and presented inconsistently across sessions, which required extended validation to accurately isolate the root cause. As an interim workaround, users were advised to perform a browser hard refresh or log out and back in, which temporarily restored normal functionality. **Root Cause:** The issue was caused by unintended behaviour in how session timeout configuration was handled during application startup and deployment. Serraview relies on a session timeout setting, retrieved from the database, to determine how long a user session remains active. In cases where the application was unable to retrieve this configuration at startup, it defaulted to a fallback session timeout value of approximately one minute instead of the intended duration. Further investigation confirmed that this fallback value could be written into the application’s cache during the build process, at a stage where database access was not available. As a result, the incorrect timeout value became temporarily embedded in the deployed application. When application instances were deployed shortly after build completion, they continued using this cached fallback value instead of retrieving the correct configuration from the database. This led to sessions expiring significantly earlier than expected, resulting in intermittent search failures, unresponsive modules, and application errors after inactivity. This also explains why the issue appeared intermittent. The incorrect timeout value was only present under specific timing conditions related to deployment and cache validity. In cases where the cache had expired or the application restarted and successfully retrieved the correct configuration, the issue did not occur. This is why actions such as refresh, re-login, or restart temporarily restored functionality. **Remediation:** A corrective change was implemented to remove reliance on the cached session configuration during application startup. This ensured that application instances consistently retrieve the correct session timeout configuration instead of using fallback values created during the build process. Following deployment of this fix: * session behaviour aligned with expected configuration * search and navigation functionality returned to normal * no recurrence of the issue was observed after periods of inactivity **Timeline \(UTC\):** March 23, 2026 10:30 AM UTC - First customer case reported around slowness and intermittent Search Functionality issues in Spaces & VBS 11:08 AM UTC - Issue acknowledged and initiated investigation. Issues replicable. 12:47 PM UTC - Issue highlighted to CloudOps team. 01:50 PM UTC - Additional cases reported indicating broader impact. 03:21 PM UTC - CloudOps team invoked Engineering team with their analysis for further investigation. 03:25 PM UTC - Jira marked as CS escalated for expedite resolution and potential HotFix. March 24, 2026 06:33 AM UTC - Root Cause identified. 09:55 AM UTC - The fix was completed and moved for QA testing. 07:18 PM UTC - Hotfix deployed to UAT environment. Additional interim checks and validation performed. March 25, 2026 11:12 AM UTC - Hotfix deployment completed in EU region for production. 1:01 PM UTC - Hotfix deployed in Prod for all region customers. 4:47 PM UTC - Support started seeking confirmation from clients. 10:47 PM UTC - First Customer Confirmation post hotfix released. **Total Duration of Event:** 54 Hours **Preventive Actions:** To prevent recurrence, changes have been implemented to ensure that: * session timeout configuration is no longer dependent on cached values generated during build or deployment * fallback configuration scenarios do not persist into runtime application behaviour * application startup reliably aligns with intended session configuration These changes remove the conditions under which incorrect session timeout values could be applied and ensure consistent and predictable session behaviour across environments.