Ooma experienced a major incident on July 30, 2020 affecting San Jose POP and Management Portal, lasting 5h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 30, 2020, 01:26 PM UTC
We are currently experiencing issues with our San Jose POP. Our team is currently investigating the issue. We will update once we have more information.
- identified Jul 30, 2020, 02:05 PM UTC
We have identified an issue with our San Jose POP. Our team is currently working to resolve the issue. Failover to the nearest POP should happen for the majority of users. You may experience calls talking slightly longer to connect during this time. If you are experiencing issues, please send us an email at [email protected].
- monitoring Jul 30, 2020, 03:02 PM UTC
A fix has been implemented and we are continuing to monitor. All traffic has now moved back to SJC.
- resolved Jul 30, 2020, 06:30 PM UTC
This morning at 05:34 PT our San Jose data center lost network connectivity and became unreachable from the internet. Working with our network operations team, we were able to identify that our equipment was online and operational, and the problem was centric to a BGP routing problem whereby our data center provider was no longer announcing our IP block to the internet. We immediately reached out to the data center operator to engage their network engineering team who were able to restore service at 07:42 PT. We are still waiting for a root cause analysis from the provider, and will be following up to build a plan to mitigate a scenario like this from occurring in the future.