[RE] Partial Degraded Service - Permanent Fix Deployed, Monitoring

Update timeline

1. investigating Jul 24, 2025, 09:11 AM UTC

   Incident Summary: We have confirmed an issue where communication with devices has become unstable, causing devices to transition to a “No response” status, DM tasks not being executed, or synchronization between the PS site and device information failing. Root Cause: We have identified the root cause of the issue as improper handling of client certificates when traffic passes through the load balancer (ingress), which resulted in certificate errors and intermittent failures in establishing communication with the cloud under high connection volumes. Resolution: Based on the root cause analysis, we are proceeding with a replacement of the load balancer (ingress) component responsible for client certificate handling. This change is expected to fundamentally resolve the issue. Until deployment is completed, we continue to restart affected processes upon detection of certificate errors to promote recovery and minimize impact.

2. identified Jan 30, 2026, 06:00 AM UTC

   We are currently implementing additional countermeasures to address the root cause and testing their effectiveness. As soon as a concrete countermeasure is finalized, we will promptly provide an additional update to customers.

3. identified Jan 30, 2026, 06:01 AM UTC

   We are continuing to work on a fix for this issue.

4. identified Feb 06, 2026, 04:34 AM UTC

   We apologize that some functions may not operate as expected due to the ongoing incident. Device communication may become unstable, leading to “No response” status, DM tasks not executing, and synchronization failures between the PS site and device information. As a temporary measure, we are restarting affected processes when errors occur to restore service stability. We will share further updates as soon as a permanent fix is prepared.

5. identified Feb 13, 2026, 05:49 AM UTC

   No instances of unstable device communication have been observed in EU production environment during this week. Monitoring and verification toward a permanent fix are ongoing.

6. identified Feb 20, 2026, 10:23 AM UTC

   Service Stability Status Over the past 14 consecutive days, we have not observed any instances of unstable device communication related to client certificate handling in the EU production environment. While work toward deploying the permanent fix is ongoing, the current service state remains stable, and no widespread recurrence has been detected in day-to-day operations. We continue proactive monitoring and validation to ensure that this stability is maintained. Background and Ongoing Work This issue was caused by the way client certificates were handled when traffic passed through the load balancer (ingress) of the cloud environment under high connection volumes in the EU environment. Through investigation and evaluation, we have confirmed that replacing the load balancer (ingress) component responsible for client certificate handling is an effective approach to resolving this behavior at its root. Because this issue manifests only at scale, validating the fix requires extended testing with a large number of simulated devices. Our focus is on ensuring that the load balancer (ingress) replacement provides sustained stability without introducing regressions. Next Steps We are currently validating the load balancer (ingress) replacement configuration that has demonstrated stable behavior in our evaluation environment. Before deploying this change to production, we are expanding test coverage to confirm: - stable and correct client certificate forwarding - no regression in device communication, monitoring, or task execution We will update this status page once these validations reach a clear conclusion, including confirmation of deployment readiness.

7. identified Mar 06, 2026, 03:12 PM UTC

   Service Stability Status For more than 4 consecutive weeks, no instances of unstable device communication caused by client certificate handling have been observed in the EU production environment. With the temporary operational measures and ongoing monitoring in place, the service continues to maintain a stable state. Validation Progress As part of validating the effectiveness of the permanent fix, we are conducting verification from the following perspectives: - Stability of client certificate forwarding under high connection volumes - No adverse impact on device communication, monitoring functionality, or DM task execution These validations have now entered the final confirmation phase, with the goal of ensuring safe deployment to the production environment. Next Update Once validation is complete and a determination can be made regarding deployment to the production environment, we will provide an update on this status page.

8. monitoring Mar 24, 2026, 10:31 AM UTC

   Service Stability Status We have completed the deployment of the permanent fix addressing the client certificate handling issue in the EU production environment. Following the deployment, the service has remained stable with no recurrence of unstable device communication.

9. monitoring Apr 01, 2026, 05:09 AM UTC

   Since the deployment of the permanent fix, no instances of unstable device communication caused by client certificate handling have been observed in the EU production environment. The service is currently operating stably, and we continue to monitor the situation to ensure sustained stability.

10. monitoring Apr 09, 2026, 06:24 AM UTC

    Service Stability Status No instances of unstable device communication have been observed in the EU production environment for 2 weeks following the release of the permanent fix. The service continues to operate stably with ongoing monitoring.

11. monitoring Apr 16, 2026, 10:07 AM UTC

    Service Stability Status No instances of unstable device communication have been observed in the EU production environment for 3 weeks following the release of the permanent fix. The service continues to operate stably with ongoing monitoring.

12. resolved Apr 24, 2026, 06:24 AM UTC

    Following the deployment of the fix, monitoring has been completed. The service remained stable throughout the monitoring period, with no recurrence of the issue observed.