Ooma experienced a critical incident on April 14, 2020 affecting San Jose POP, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 14, 2020, 06:10 PM UTC
We are currently investigating a possible outage with our SJC POP. Updates to follow shortly.
- investigating Apr 14, 2020, 06:25 PM UTC
We have a confirmed outage on our SJC impact. We are still investigating the cause. We expect phones and calls in that region to failover to our VAN POP shortly.
- monitoring Apr 14, 2020, 07:51 PM UTC
At approximately 10:55am PT, a service provider (Telia Networks) that is one of many networks that provides internet connectivity into our facility in San Jose started to experience significant latency and packet loss that resulted in network links coming up and down repeatedly (also known as flapping). As a precautionary measure, our network operations team took our San Jose proxies out of rotation, which would have caused clients attached to our network to fail over to the next closest site in Vancouver. Any device attached to our network that was provisioned using our device provisioner will automatically do this. Some customers that have provisioned their devices manually may need extra guidance on how to configure fail-over settings correctly. If your device was offline during this window, please reach out to [email protected] to have your configuration reviewed and updated. Finally, at approximately 12:07pm PT, we were able to make appropriate modifications to our routing infrastructure in San Jose to take Telia Networks out of the network path, which restored connectivity there without packet loss or latency.
- resolved Apr 15, 2020, 06:10 PM UTC
Our monitoring over the last 24 hours has shown no more instability in the network after Telia was removed yesterday. We are considering this outage resolved.