PubNub incident
Presence is experiencing elevated latencies and error rates
PubNub experienced a notice incident on September 2, 2025 affecting North America Points of Presence and Asia Pacific Points of Presence and 1 more component, lasting 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Sep 02, 2025, 06:34 PM UTC
At about 11:09, the Presence began to experience elevated latencies and error rates. PubNub Technical Staff is investigating and more information will be posted as it becomes available. If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support ([email protected]).
- resolved Sep 02, 2025, 06:50 PM UTC
The issues affected the Presence service from 18:08 to 18:16 UTC. We will follow up with a post-mortem soon. We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support ([email protected]) if you wish to discuss the impact on your service.
- postmortem Sep 05, 2025, 10:27 PM UTC
### **Problem Description, Impact, and Resolution** At 18:09 UTC on September 2, 2025 we observed increased error rates and latency for our Presence service in our San Jose, Virginia, and Tokyo regions. We increased capacity in those regions and the issue was resolved at 18:16 UTC. This issue occurred because a bug in one of our APIs allowed a request to execute an operation that exceeded assumed limits in extreme cases. In this case, a large number of such requests were executed that resulted in out-of-memory conditions for the Presence service. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future we have enforced the intended limit on the API in question. We have also added additional testing and monitoring in this area.