Elevated Subscribe Latency and Errors in US-East

Update timeline

1. investigating Oct 30, 2023, 06:11 PM UTC

   Starting around 13:45 UTC, we began observing elevated latency and errors in calls made to our Subscribe endpoint in our US East point of presence. PubNub Technical Staff is investigating, and more information will be posted as it becomes available. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support ([email protected]) if you wish to discuss the impact on your service.

2. identified Oct 30, 2023, 06:31 PM UTC

   The issue has been identified and a fix is being implemented. We will continue to provide regular updates.

3. resolved Oct 30, 2023, 06:59 PM UTC

   This incident has been resolved with no errors observed for the last 30 minutes. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support ([email protected]) if you wish to discuss the impact on your service. An RCA will be provided soon.

4. postmortem Oct 31, 2023, 03:59 PM UTC

   ### **Problem Description, Impact, and Resolution** On October 30, 2023 at 17:52 UTC, we observed a high number of errors and increased latency in our Subscribe service, mostly in our US-East point of presence. This was a result of increased Subscribe traffic in the region growing since around 13:45 UTC. Customers making subscribe API calls at the time may have experienced increased response latency, or failures with error. We increased service capacity, and memory allocation in all regions, and the issue was resolved on October 30, 2023 at 18:23 UTC. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future, we updated our monitoring to provide early warning of increased Subscribe traffic, giving us sufficient time to scale before issues occur.