PubNub incident
Elevated rate of latency in India Point of Presence
PubNub experienced a minor incident on April 21, 2023 affecting Southern Asia Points of Presence, lasting 1h 20m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 21, 2023, 01:44 PM UTC
Starting at 12:05 UTC, Customers using our India PoP might be experiencing spikes of increased latencies across multiple services.
- identified Apr 21, 2023, 02:12 PM UTC
The issue has been identified and we have not seen latency spike in the past 30 minutes.
- monitoring Apr 21, 2023, 02:31 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 21, 2023, 03:05 PM UTC
This incident has been resolved with no spikes observed since around 13:30 UTC.
- postmortem Apr 25, 2023, 02:28 PM UTC
### **Problem Description, Impact, and Resolution** On April 21, 2023 at 12:04 UTC, we observed increased latency for Publish API calls to our Mumbai Point of Presence. The issue specifically affected some customers whose publish calls are routed through the PubNub Functions service. We scaled up capacity, and the issue was resolved on April 21, 2023 at 13:40 UTC. ### **Mitigation Steps and Recommended Future Preventative Measures** To prevent a similar issue from occurring in the future, we are implementing changes to our scaling process to better address these traffic patterns and adding improved alerting and monitoring for earlier warning of such issues. This should assist with faster responses if there are repeated intervals of high load then lower load.