PubNub Outage History

**Problem Description, Impact, and Resolution** Starting at 16:46 UTC on Nov. 20, 2025, we noticed a small number of errors with the publish API in the North American and Asia Pacific regions. The system automatically recovered with all functionality fully restored by 16:50 UTC on Nov. 20, 2025.

### **Problem Description, Impact, and Resolution** At 18:14 UTC on September 7, 2025 we observed increased error rates and latency for our Presence service in our San Jose, Virginia, and Tokyo regions. We increased capacity in those regions and the issue was resolved at 18:17 UTC. This issue was a recurrence of the issue [experienced on September 2, 2025](https://status.pubnub.com/incidents/1n8xk6w5y9lk), where a bug in one of our APIs allowed a request to execute an operation that exceeded assumed limits in extreme cases, causing out-of-memory conditions for the Presence service. ### **Mitigation Steps and Recommended Future Preventative Measures** In the previous instance of this issue, we placed restrictions on the API in question; those changes were not restrictive enough, which allowed for this recurrence. We have corrected that oversight, as well as increased memory capacities in this area of our system as an additional safeguard.

### **Problem Description, Impact, and Resolution** At 18:09 UTC on September 2, 2025 we observed increased error rates and latency for our Presence service in our San Jose, Virginia, and Tokyo regions. We increased capacity in those regions and the issue was resolved at 18:16 UTC. This issue occurred because a bug in one of our APIs allowed a request to execute an operation that exceeded assumed limits in extreme cases. In this case, a large number of such requests were executed that resulted in out-of-memory conditions for the Presence service. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future we have enforced the intended limit on the API in question. We have also added additional testing and monitoring in this area.

### **Problem Description, Impact, and Resolution** On July 8, 2025 at 17:15 UTC on, we observed elevated errors and increased latency for customers using our Presence service in the US East, US West, and Tokyo regions. We identified an abnormal concentration of traffic and applied a configuration change to redistribute load across our infrastructure. The issue was resolved by 17:45 UTC the same day. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future, we have implemented targeted balancing for Presence traffic patterns that exhibit such a concentrated load. This allows us to distribute traffic more evenly across infrastructure components. In the coming days, we will work to identify additional patterns that may require similar configuration changes to ensure even load balancing.

### Problem Description, Impact, and Resolution Starting at 12:20 pm UTC on June 2, 2025, we noticed an increase in errors and latency affecting the Presence service in multiple regions. After investigating the increase in errors and latency, we identified the cause to be a higher-than-normal traffic pattern. In response, we scaled up resources to handle the request load, and the Presence service was fully restored in the affected regions by 1:51 PM UTC on June 2, 2025. ### Mitigation Steps and Recommended Future Preventative Measures To prevent a similar issue from occurring in the future, we are tuning our autoscaling configuration, as well as analyzing system behavior and bottlenecks observed during the incident window to inform further adjustments.

### **Problem Description, Impact, and Resolution** On May 23, 2025 at 20:51 UTC, PubNub experienced increased errors and latency for a subset of traffic within a single availability zone in the US East region. The root cause was an operational error during an infrastructure change. We unintentionally drained traffic from newly deployed load balancers. As a result, the system routed traffic incorrectly, causing elevated error rates and latency for some users. Once the issue was identified, traffic was rerouted, and service performance returned to normal levels. The incident was resolved by 21:04 UTC the same day. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent the issue from recurring in the future, the change procedure has been updated to reference isolated, non-production resources during preparation stages. If run prematurely, the script will now operate on an empty set, ensuring no production traffic is impacted.

### **Problem Description, Impact, and Resolution** On May 16, 2025 at 09:05 AM UTC, some customers in US and APAC regions experienced elevated error rates and increased latency when using the Presence service. This disruption was caused by an interruption of underlying infrastructure supported by a third-party provider service. This failure also led to delays in resolving the issue, which prolonged the impact. We were able to adjust resource capacity to overcome this and bring the service back to normal. Services were fully restored on May 16, 2025 at 10:23 AM UTC. ### **Mitigation Steps and Recommended Future Preventative Measures** We increased memory on the Presence pods and are working with the third-party provider to address the issues encountered during the incident. To prevent such issues from recurring in the future, we will also work to improve the Presence service to be more resilient to fluctuations in traffic.

## **Problem Description, Impact, and Resolution** On May 12, 2025 at 23:22 UTC, we observed elevated errors and increased latency for customers using our Presence service in the Virginia \(US-East\), California \(US-West\), and Tokyo \(AP-North\) regions. After discovering the source of the errors and latency, a solution was set up and ready to be implemented, but the incident was resolved before it was implemented. This issue occurred because we could not be sure that the fix would not negatively impact some functionality of the presence system. ## **Mitigation Steps** To prevent a similar issue from occurring in the future, we are ensuring that all presence features work with channel sharding to safely enable the solution to any sudden usage changes.

### **Problem Description, Impact, and Resolution** On March 10, 2025, we observed elevated latency and errors in the Presence service across our global points of presence from 18:26 to 18:40 UTC and again from 20:05 to 20:27 UTC. After investigating, we found that a short spike in traffic with an unusual pattern caused a downstream service provider to respond with increased latency, which affected our own response times. While this incident resolved quickly, we are working with the third-party provider to better handle this scenario in the future.

### **Problem Description, Impact, and Resolution** At 21:50 UTC on 2024-12-11 we observed increased latencies and error rates across all services in our US-East point-of-presence and, a few minutes later, in US-West as well. We observed that the PubNub Access Manager \(PAM\) was at the center of the degradation, and an investigation noted that nodes in that service were highly memory constrained. We increased capacity and the issue was mitigated in both points-of-presence at 22:10 UTC, and declared resolved at 22:22 UTC. This issue occurred because a previously unseen pattern of customer behavior overwhelmed a cache in the PAM system, causing memory to become constrained and performance to degrade. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future we changed the cache capacity and updated our monitoring to alert on this and similar patterns of behavior.

### **Problem Description, Impact, and Resolution** At 07:23 UTC on December 9th, we received alerts indicating high error levels related to storage writer operations in one of our data centers. Shortly after, one of our third-party service providers reported service disruption in their environment. The service provider began the process of replacing the affected nodes. Throughout the restoration process, we closely monitored our systems to assess how the issue impacted our environment. Once all nodes were successfully restored, error levels returned to normal, and all associated alerts were resolved. ### **Mitigation Steps and Recommended Future Preventative Measures** We have worked with our vendor to ensure that nodes of this type will be on redundant infrastructure going forward, so that there is less exposure to this kind of incident.