Elevated latencies in the EU PoP Subscribe Service

Update timeline

1. resolved Jun 01, 2023, 04:41 PM UTC

   On June 1, 2023 between 08:18 and 09:06 UTC, customers using Subscribe service in our EU Central PoP may have experienced some errors, timeouts, and delays in message delivery. We will publish a root cause analysis soon.

2. postmortem Jun 05, 2023, 04:43 PM UTC

   ### **Problem Description, Impact, and Resolution** On June 1, 2023, we observed two subscribe latency spikes in our Europe Point of Presence. The spikes occurred from 08:18 AM UTC through 09:06 AM UTC, and from 09:44 AM UTC through 09:54 AM UTC. During these times, users in the Europe region may have experienced slower than normal responses on subscribe calls. The higher than normal latency affected one of multiple access zones in the region. Shortly after detecting the increase in latency, the cause of the issue was identified, and a fix was deployed, restoring the region to normal operational status by 09:54 AM UTC on June 1. This issue occurred when a code deployment to the region overwrote a configuration previously deployed, resulting in a lack of resources in the access zone. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future, we are applying a fix to all clusters. Additionally, we are improving alerting around publish to subscribe latency so we are quickly notified if a similar issue were to occur.