PubNub incident
Elevated Channel Groups Subscribe latency in the US-West PoP
PubNub experienced a minor incident on August 12, 2023 affecting Publish/Subscribe Service and North America Points of Presence, lasting 1h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 12, 2023, 01:05 AM UTC
On August 11 at 23:45 UTC, we began to observe increased latency for channel group subscribes in our US-West PoP, which could result in delays in receiving messages. PubNub Technical Staff is investigating, and more information will be posted as it becomes available. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support ([email protected]) if you wish to discuss the impact on your service.
- identified Aug 12, 2023, 01:33 AM UTC
We believe the issue has been identified and a fix is being implemented. We will provide updates as they become available.
- monitoring Aug 12, 2023, 01:43 AM UTC
This issue has been resolved and latency has returned to normal levels. We will continue to monitor services for the next 30-60 minutes. We will continue to provide updates here.
- monitoring Aug 12, 2023, 02:34 AM UTC
We are continuing to monitor for any further issues for the next 30-60 minutes. We will continue to provide updates here.
- resolved Aug 12, 2023, 02:55 AM UTC
This incident has been resolved. The incident was declared with an overabundance of caution, and was determined to be limited to a handful of customers. Please contact PubNub Support ([email protected]) if you wish to discuss the incident.
- postmortem Aug 31, 2023, 07:15 PM UTC
### **Problem Description, Impact, and Resolution** On Friday, August 11th at 23:45 UTC, we observed a delay in message delivery for subscribe requests using our Channel Groups service. After identifying the delay, we restarted the affected pods, and the issue was resolved at 01:44 UTC on Saturday, August 12th. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future, we are improving the Channel Groups service communication, as well as exploring enhanced error handling and retries to ensure improved monitoring and alerting.