PubNub incident
Increased latency and errors observed in US-West
PubNub experienced a minor incident on October 20, 2025 affecting Publish/Subscribe Service and North America Points of Presence and 1 more component, lasting 10h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 20, 2025, 01:37 PM UTC
Beginning at 12:40 UTC, we observed increased latency and errors in our US-West region for multiple services. PubNub Technical Staff is currently investigating, and more updates will follow once available.
- investigating Oct 20, 2025, 01:47 PM UTC
We are continuing to investigate this issue.
- investigating Oct 20, 2025, 02:21 PM UTC
Apologies, but we are continuing to investigate this issue.
- identified Oct 20, 2025, 02:41 PM UTC
The issue has been identified and a fix is being implemented.
- identified Oct 20, 2025, 03:13 PM UTC
We are still working with our infrastructure provider to mitigate the effects of the ongoing issues. We have been monitoring and handling traffic to minimize disruptions. While points of presence in the US may see slightly higher latencies, we are stable. Please reach out to PubNub support to report issues.
- identified Oct 20, 2025, 04:43 PM UTC
We are still stable, but seeing minor issues. We are still working with our infrastructure provider to address the remaining issues. Please reach out to PubNub support with any issues.
- monitoring Oct 20, 2025, 05:25 PM UTC
PubNub remains stable and fully operational. Our infrastructure provider is continuing to apply mitigation steps to restore normal health across their network and related services. We are closely monitoring the situation and coordinating with them to ensure continued stability. Please reach out to PubNub Support if you experience any service issues.
- identified Oct 20, 2025, 05:25 PM UTC
The issue has been identified and a fix is being implemented.
- identified Oct 20, 2025, 06:00 PM UTC
PubNub continues to remains stable, with only minor issues being observed. Our infrastructure provider is making progress in restoring their services, with early signs of recovery in several regions. We continue to work closely with them as they complete mitigations across the remaining areas. Please reach out to PubNub Support if you experience any impact.
- identified Oct 20, 2025, 06:37 PM UTC
PubNub remains stable, with only minor issues being observed. Our infrastructure provider is making progress in restoring their services, with early signs of recovery in several regions. We continue to work closely with them as they complete mitigations across the remaining areas. Please reach out to PubNub Support if you experience any impact.
- identified Oct 20, 2025, 07:06 PM UTC
Our status is the same: stable, and working with our infrastructure provider to restore services to their normal state. Our normal policy is to update every 30 minutes, but we will stop issuing this status; we will update this page when a change to the status occurs. As always, reach out to PubNub support with any issues.
- monitoring Oct 20, 2025, 08:25 PM UTC
PubNub remains stable, and our infrastructure provider continues to make steady progress in restoring services. System performance and reliability are improving, and functionality across affected components is returning to normal levels. We continue to monitor closely and will share additional updates as recovery progresses. We will provide another update when there are any changes in status which we expect within the hour. Please contact PubNub Support if you experience any impact.
- monitoring Oct 20, 2025, 10:32 PM UTC
PubNub remains stable, and our infrastructure provider has restored normal service capacity. Systems are now operating at pre-event levels, and most dependent services are performing as expected. We continue to monitor closely and will share further updates as needed. Please contact PubNub Support if you experience any impact.
- resolved Oct 20, 2025, 11:48 PM UTC
This incident has been fully resolved. Our infrastructure provider has restored normal operations, and all systems are functioning as expected. Service performance and reliability have returned to pre-event levels. PubNub continues to monitor systems closely to ensure ongoing stability and we will provide an RCA in in the next couple of day. Thank you for your patience and understanding throughout this event. If you experience any further issues, please reach out to PubNub Support.
- postmortem Oct 27, 2025, 06:33 PM UTC
### **Problem Description, Impact, and Resolution** On October 20th, 2025 at 07:06 UTC, our monitoring systems alerted us to elevated error levels across multiple PubNub services in the IAD region \(US-East\). Some customers may have experienced increased error rates and latency, as well as intermittent issues with Presence service availability across IAD \(US-East\), SJC \(US-West\), and HND \(AP-Northeast\). We quickly determined the issue was caused by a broader infrastructure outage affecting our cloud provider \(AWS\) in the IAD region. We initiated regional failover procedures and re-routed new connections to alternate regions. However, due to undefined steps in some of our failover processes and delays accessing some tools due to the provider issue, existing connections for some services remained degraded for longer than expected. To restore full service, we manually reset established connections, re-routed Presence traffic to Frankfurt \(EU-Central\), and brought on additional infrastructure in other regions to absorb traffic. Errors were mitigated by 09:20 UTC. Later in the day, additional regional load in US-West triggered a new wave of service degradation. We responded by isolating the US-East region again and scaling up balancer capacity in US-West. PubNub services were stabilized by 13:20 UTC, and remained in a monitoring state while our infrastructure provider worked to fully resolve the underlying issue. By 22:35 UTC, our provider reported full restoration of service. After validating stability in US-East, we completed rebalancing traffic by 23:48 UTC, and declared the incident resolved. ### **Mitigation Steps and Recommended Future Preventative Measures** While this incident was caused by an external infrastructure outage, we’ve identified several opportunities to strengthen our internal readiness and response procedures. We are consolidating and centralizing our regional failover procedures to ensure they are immediately accessible and complete for all production services. Any gaps in our process documentation for newer services will be addressed to ensure readiness before they are fully adopted into production. Additionally, we are reviewing and resolving issues with internal tooling, including inventory and DNS resolution problems, which made mitigation more difficult during the incident. These improvements will ensure faster and more consistent responses to future infrastructure-level disruptions, and reduce potential impact on customer traffic across regions.