Ooma experienced a notice incident on October 31, 2018 affecting Vancouver POP and San Jose POP, lasting 1h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 31, 2018, 11:27 PM UTC
We are currently investigating an issue where some customers are experiencing intermittent busy signals when placing both inbound and outbound calls.
- investigating Oct 31, 2018, 11:32 PM UTC
We are continuing to investigate reports of delayed call completion in both VAN and SJC POPs.
- identified Oct 31, 2018, 11:37 PM UTC
We have resolved intermittent busy and delayed calls in our VAN POP. Issues still remain for customers connected to SJC. We are implementing a fix as quickly as possible.
- identified Oct 31, 2018, 11:39 PM UTC
We are continuing to work on repairing issues in SJC. Calls are experiencing delays in delivery for both inbound and outbound.
- identified Oct 31, 2018, 11:42 PM UTC
We have identified the source of the issue in our SJC POP. We are working on implementing a fix as soon as possible.
- identified Oct 31, 2018, 11:50 PM UTC
Calls are being delivered but some calls still are experiencing delays for customers in SJC. We are continuing to implement a fix.
- identified Oct 31, 2018, 11:50 PM UTC
We are continuing to work on a fix for this issue.
- identified Nov 01, 2018, 12:07 AM UTC
We are continuing to repair delays to call delivery in SJC POP.
- monitoring Nov 01, 2018, 12:20 AM UTC
We have implemented a fix and calls are delivering to customers connect to SJC. We are monitoring the situation and will post here with updates as needed or when we believe the issue to be resolved.
- resolved Nov 01, 2018, 01:16 AM UTC
We believe this incident to be resolved as there have been no further delays to call delivery both inbound in outbound across our network. We will provide a post mortem to this even within the next 24 hrs.
- postmortem Nov 02, 2018, 12:04 AM UTC
We have completed our investigation of service interruption in our San Jose and Vancouver POPs for October 31, 2018. We were monitoring an issue with one our servers in our SJC data centre. At 4:06PM Oct. 31 we removed that server from the availability pool because it was causing timeouts that were delaying requests. This was done to ensure health of the entire network. When the connections failed over to the backup facility, the congestion created delays answering requests in that facility as well. We moved traffic back to the primary in SJC and repaired the issue. The total service interruption in VAN was approximately 5 minutes. The total service impact in San SJC was 52 minutes; calls were being created and delivered with delays for 36 minutes of the impacted time. The root cause of the issue has been identified and we have adjusted configurations in SJC and VAN to allow for a more graceful failover between the two sites. We apologize for the impact this may have caused. We know that voice services are a critical part of your business and Voxter is proud to be your provider.