Proxyclick experienced a critical incident on September 15, 2024 affecting Dashboard and iPad app and 1 more component, lasting 1h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 15, 2024, 03:30 PM UTC
We are currently investigating an issue with Proxyclick. We will update you when we have more information.
- investigating Sep 15, 2024, 03:34 PM UTC
We are currently investigating reports of user authentication failures
- investigating Sep 15, 2024, 03:53 PM UTC
We are continuing to investigate this issue.
- monitoring Sep 15, 2024, 04:05 PM UTC
We have implemented a solution for the issue affecting Proxyclick Application and are currently monitoring the situation to ensure stability and performance. Our Engineering team is overseeing the process to confirm that the issue has been fully resolved
- monitoring Sep 15, 2024, 04:06 PM UTC
We are continuing to monitor for any further issues.
- resolved Sep 15, 2024, 05:12 PM UTC
We are pleased to share that this incident is resolved. We will publish our root cause analysis findings on this incident within 10 business days.
- postmortem Sep 27, 2024, 04:35 PM UTC
**Type of Event:** S1: Visitor app down for users - Error: "Something went wrong while trying to validate your credentials" **Services/Modules Impacted:** Visitor app login and related applications **Root Cause:** Our Infra team discovered that the service principal, essential for authenticating access to critical services, had expired. As a result, the certificate in the gateway was not updated. **Remediation:** The service principal was updated manually which successfully restored service functionality. **Timeline:** _All times listed in CEST_ 16:33 - Received alert indicating a service issue. 16:35 - Infra Team started investigating the alert. 16:50 - Root cause was identified. 16:51 - First client reported issue. 17:30 - Fire alarm triggered. 17:41 - The service principal was updated, and the issue was resolved at Infra level. 18:05 - Status page updated to monitoring as the issue was no longer observed. 19:12 - Status page updated to resolved. **Total Duration of Event:** 2 hours 21 minutes **Preventive Action:** To proactively enhance our service reliability, our Infra team is implementing an automated alert system for service principal expirations. This system will provide advance notice, facilitating timely renewals and helping to ensure uninterrupted service, aligning with our commitment to seamless service delivery.