Elevated rate of latency and timeouts in US East Point of Presence (POP)

Update timeline

1. identified Apr 02, 2023, 04:19 PM UTC

   Beginning at 15:34 UTC 04/02, some customers may be experiencing a higher rate of timeouts and errors in Presence, Push, HIstory, Subscribe, and Auth Services.

2. monitoring Apr 02, 2023, 04:39 PM UTC

   Around 16:30 UTC a fix has been implemented and we are monitoring the results for next 30 mins.

3. resolved Apr 02, 2023, 05:13 PM UTC

   This incident has been resolved and we will follow up with a post-mortem soon. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support ([email protected]) if you wish to discuss the impact on your service.

4. postmortem Apr 05, 2023, 12:33 AM UTC

   ### **Problem Description, Impact, and Resolution** On April 2, 2023 at 15:34 UTC we observed an increased rate of timeouts and errors in our services for Presence, Push, History, Subscribe, and Auth Services in our US East Point of Presence. We identified the issue was related to repair work being performed on this PoP by one of our third party vendors. After identifying the issue, we rerouted traffic from US East to US West and moved the incident to Monitoring status since improvement was observed. Once we received an update from our third-party vendor that the repair work was complete, we routed traffic back to our US East PoP, and the issue was resolved at 17:13 UTC. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future we will implement more granular monitoring and alerting for early warning of such issues to the affected infrastructure.