Minor Issue - LXP EU Prod 504 Gateway Timeout Error

Update timeline

1. resolved Jun 22, 2026, 09:24 AM UTC

   The CSOD Technology Team observed a performance degradation affecting production portals across EU. The problem began at 10:35 PM PST and was resolved at 11:30 PM PST. During this time, clients could not enter LXP portals and encounter 504 Gateway Timeout Error.

2. postmortem Jul 10, 2026, 06:32 AM UTC

   **Incident Summary:** On 22nd June 2026, users in the EU region experienced intermittent service degradation caused by DNS resolution failures. Our LXP applications were unable to reliably resolve internal service names, resulting in connectivity issues and increased response times. A subset of customers experienced HTTP 504 Gateway Timeout errors during the incident window. **Impact:** A subset of EU customers experienced service degradation, including increased latency and 504 Gateway Timeout responses. Internal application communication was intermittently disrupted due to failed DNS lookups. Customer transactions and requests dependent on affected services were delayed or unsuccessful until DNS functionality was restored. **Root Cause:** The issue was caused by resource saturation and degraded performance of DNS within the EU cluster. As DNS response times increased, application pods experienced delays and failures when resolving internal and external hostnames. This impacted service-to-service communication and request processing across the environment. **Corrective Actions:** As an immediate mitigation, additional resources were allocated to the DNS pods to handle the sudden spike in DNS traffic and restore service stability **Preventive Actions:** The team is re-evaluating DNS configuration and scaling parameters, reviewing DNS traffic patterns, and adjusting resource allocations based on updated capacity planning.