S2- Elevated response times in the admin dashboard

Update timeline

1. investigating Nov 18, 2024, 01:51 PM UTC

   Our Infra team is currently investigating reports of increased loading times and slow response in the Proxyclick admin dashboard.

2. investigating Nov 18, 2024, 02:45 PM UTC

   We are continuing to investigate this issue.

3. resolved Nov 18, 2024, 04:22 PM UTC

   We are pleased to share that this incident is resolved as we have confirmed full restoration of admin response time performance. We will publish our root cause analysis findings on this incident within 10 business days. Thank you for your continued patience.

4. postmortem May 19, 2025, 08:03 AM UTC

   We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Type of Event:** S2 – Visitor \(PXC\) API and Dashboard Performance Issues **Services/Modules Impacted:** Visitor Dashboard, web application, and API **Issue Summary/Background:** An increase in API response time and a significant spike in HTTP 500 errors were observed. Upon investigation, the logs revealed a rise in MySQL deadlocks and application server errors. Specifically, the following error was noted in the application logs: _\[HPM\] Error occurred while trying to proxy request /full/app/CO-DEDP117/vm/kiosk/instances from_ [_app.example.com_](http://app.example.com/) _to_ [_https://api.example.com_](https://api.example.com/) _\(ECONNRESET\) \(_[Errors | Node.js v23.11.0 Documentation](https://nodejs.org/api/errors.html#errors_common_system_errors)) ‌ These issues affected the overall application performance and user experience. **Root Cause:** High latency and increased MySQL deadlocks. **Remediation:** While reviewing MySQL and application server logs, the team identified an increase in deadlocks in MySQL logs and application connection resets. To mitigate the issue, the Eptura CloudOps team performed a database. The restart helped in returning the application functionality to normal. We also noticed that API response times significantly improved and the HTTP 500 errors decreased substantially. **Timeline:** _All times listed in CEST_ _9:22 a.m: A large spike in 5xx HTTP response errors_ and _elevated response times from API endpoints were observed._ 2:56 _p.m.: An incident was reported, prompting Eptura to initiate the investigation._ _3:51 p.m: Eptura updated the Visitor status page to reflect the incident and investigation._ _4:57 p.m.: The Eptura Infra team identified the root cause of the issue and suggested that the response times have now been stable._ _5:53 p.m: The Eptura team updated cases to ask customers for initial feedback on the issue._ _6:22 p.m: The Eptura team updated the status page confirming the issue is resolved._ **Total Duration of Event:** 1 Hour 12 minutes **Preventive Actions**: To mitigate future occurrences, we have scheduled a proactive MySQL service restart every Monday before working hours to maintain database stability.