[Eptura Visitor] S2 - Elevated response times in the admin dashboard

Update timeline

1. investigating Dec 08, 2025, 02:40 PM UTC

   Our team is currently investigating reports of increased loading times and slow response in the Visitor admin dashboard.

2. identified Dec 08, 2025, 03:20 PM UTC

   We are currently reverting the recent release to maintain system stability and prevent any potential disruptions. During this process, customers may notice that new features introduced in the latest release are temporarily unavailable, but core functionality remains unaffected. Our CloudOps team has initiated the rollback and is closely monitoring progress to ensure a smooth transition. Once the rollback is complete and system performance is validated, we will provide confirmation. Thank you for your patience as we work to resolve this promptly.

3. monitoring Dec 08, 2025, 07:26 PM UTC

   The required rollback has been completed. Response times have gone back to a normal expected level. We are continuing to monitor this situation and will provide additional updates as they become available. Thank you for your patience as we work to resolve this issue.

4. investigating Dec 09, 2025, 10:41 AM UTC

   We are aware that the issue has resurfaced, resulting in elevated response times. Our internal team is actively investigating the cause and working to restore normal performance as quickly as possible. We will share further updates as soon as more information becomes available. Thank you for your patience.

5. identified Dec 09, 2025, 01:43 PM UTC

   Our systems are up and running, and the database is healthy. However, we noticed slower API response times due to a configuration limit. We’ve restarted key services to improve performance. We also found an issue in the SCIM service that caused the disruption. Our engineering team is working on the fix. Thank you for your patience as we work to restore full functionality.

6. identified Dec 09, 2025, 03:11 PM UTC

   We have identified the root cause and are now applying a hotfix. The deployment is expected to take approximately one hour. During this time, some services may be temporarily unavailable. We will provide an update once the hotfix has been successfully applied and validated.

7. identified Dec 09, 2025, 08:52 PM UTC

   Investigating – We are continuing to experience degraded performance and intermittent issues accessing the site. Earlier today, we implemented a hotfix, but it did not deliver the expected improvement. Our teams are actively investigating the root cause and working to restore normal performance as quickly as possible. We will share further updates as soon as more information becomes available. Thank you for your patience while we work to restore normal performance.

8. investigating Dec 10, 2025, 12:00 AM UTC

   We are still seeing degraded performance and intermittent access issues on the site. The hotfix applied earlier did not provide the expected improvement. Our teams remain focused on identifying the root cause and implementing a solution to restore normal performance as quickly as possible. We will continue to share updates as soon as we have more information. Thank you for your continued patience while we work to resolve this issue.

9. investigating Dec 10, 2025, 09:27 AM UTC

   Our engineering teams continue to closely monitor the elevated query activity and the resulting performance impact on the site. They are actively analysing system behaviour and validating several paths to stabilise performance. Additional diagnostics are underway, and we will provide further updates as soon as more information becomes available. Thank you for your continued patience as we work to ensure full service reliability.

10. investigating Dec 10, 2025, 07:23 PM UTC

    Our teams continue to actively work on this issue and have escalated it to third-party vendors for additional support. The root cause remains unknown at this time. As a mitigation step, we are adding additional infrastructure. These changes are significant and require careful execution to ensure stability and safety. We expect these efforts to be completed by 04:30 AM UTC. We will provide the next update after that point. Thank you for your continued patience as we work to restore full service reliability

11. identified Dec 11, 2025, 08:12 AM UTC

    Our teams have identified specific database queries contributing to delays within the platform and have shared these findings with our engineering group. Work is underway on targeted improvements aimed at addressing the behavior we are seeing. We will continue to monitor closely as these changes progress and will provide further updates as more information becomes available. Thank you for your patience as we move toward full resolution.

12. monitoring Dec 11, 2025, 12:21 PM UTC

    System performance has improved after increasing API server capacity and enabling multi-core support. An additional API server has been added, and response times remain stable. We are monitoring the system closely and will share further updates as needed. Thank you for your continued patience.

13. resolved Dec 12, 2025, 02:53 PM UTC

    We are pleased to inform you that the issue with the 'admin dashboard slowness' has been resolved. Our team has completed the necessary actions and verified that the service is now functioning normally. A Root Cause Analysis (RCA) will be conducted to understand the incident in detail and will be made available on our Status Page within 10 days. Thank you for your patience and cooperation throughout this process. If you have any further questions or concerns, please feel free to reach out.

14. postmortem Jan 09, 2026, 03:55 PM UTC

    We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Type of Event:** S2 - Performance issue impacting dashboard and check-in flows **Services/Modules Impacted:** All services **Issue Summary/Background:** Between December 8 and December 10, 2025, some customers experienced slower load times and occasional timeouts when using the Eptura Visitor dashboard and related workflows. In certain cases, this affected check-in flows and badge printing. While the service remained available, performance did not meet our usual standards during peak usage periods. **Root Cause:** The issue began after a scheduled update and continued during high-traffic periods, even after the update was rolled back. Our investigation revealed that the root cause was a configuration setting that limited Eptura Visitor services to single-core processing per server. Although servers had multiple cores, Node.js processes were not using them, creating a bottleneck under heavy load. **Remediation:** To restore performance and stabilize the platform, we took immediate action: * Rolled back the recent update to reduce risk * Increased server capacity and optimized runtime settings * Enabled multi-core processing and clustering for better load distribution * Conducted a full infrastructure health review to confirm stability These steps successfully restored normal performance and improved baseline response times. **Timeline \(UTC\):** December 8, 2026 14:19 – First customer reporting of slow performance. 18:18 - Eptura support engaged the CloudOps team for investigation. 19:46 - Multiple customers impacted, case count increased. The issue severity was updated to S2. 20:35 – As part of the initial investigation, Eptura initiated a rollback of a recent update as a precautionary measure. 21:45 – The rollback was completed, and Eptura started checking with customers for feedback. 23:25 – Performance showed temporary improvement, and the incident remained under observation. December 9, 2025 03:19 – The incident was escalated internally to bring in additional teams. 19:18 – A corrective fix was deployed. 23:03 – Monitoring showed that the fix did not fully resolve the issue, and investigation continued. December 10, 2025 10:28 – Eptura Engineering and Cloud teams began joint, in-depth troubleshooting. 17:18 – Investigation was expanded to include platform and infrastructure partners. 23:43 – Additional capacity and configuration improvements were implemented to stabilize performance. December 11, 2025 10:05 – We observed improvement in system performance. 14:48 – Additional capacity was added to support sustained recovery. 17:46 – Eptura support started receiving confirmation of resolution. December 12, 2025 19:01 – Monitoring confirmed performance had returned to normal levels. December 15, 2025 11:25 – Continued monitoring confirmed long-term stability. December 22, 2025 11:51 – The incident was formally resolved and closed. **Total Duration of Event:** 75 hours **Preventive Actions:** To prevent similar incidents in the future, we are implementing the following improvements: * Validate multi-core runtime profiles and clustering for all critical Eptura Visitor services * Strengthen scaling and failover policies for peak traffic scenarios * Enhance monitoring and alerting to detect performance degradation earlier All services are operating normally, and we continue to monitor the platform closely. Thank you for your understanding as we work to deliver the reliable experience you expect.