Data Center Issues in Dallas TX

Update timeline

1. identified May 26, 2024, 12:30 PM UTC

   This morning, Sunday, May 26th, starting at 7:21 AM EST, we experienced a complete outage from our Dallas, Texas, data center. We are communicating with the data center personnel to determine what impact this outage might have and what the estimated recovery time is. Our voice services in Greensboro, NC, and also Research Triangle Park, NC, are fully operational, and no customer should experience any issues with voice or messaging services. Our geo-redundant platform works as designed, and calls routed to or from Dallas, TX, will fail over to the other two data centers.

2. identified May 26, 2024, 01:41 PM UTC

   A support engineer in Dallas, Texas, has determined we have experienced an unexpected power outage. He is working to restore power this morning in Dallas, Texas.

3. monitoring May 26, 2024, 02:17 PM UTC

   Power has been restored to our Dallas, Texas, data center. In the coming hours, we will wait for a post report from our data center https://evocative.com/data-centers/plano-tx-data-center-dal1/ It might take a few minutes for all voice and messaging services to be up and operational. We will continue to monitor.

4. resolved May 26, 2024, 02:19 PM UTC

   All services are live and operational in our Dallas, Texas, data center.

5. postmortem May 27, 2024, 06:36 PM UTC

   **Dallas Data Center Postmortem Report:** At approximately 7:00 AM EST on Sunday, May 26th, 2024, there was an issue with the utility feed at our Dallas, TX, data center. The Battery UPS systems picked up the load and attempted to transfer it to the generator. The generators failed to start properly and could not pick up the load before the UPS systems drained the batteries. At approximately 8:30 AM, they got the UPS systems back online with utility power and have been bringing services back online as quickly as possible. It was an additional one hour and forty minutes before all services in this data center were restored on the utility feed. They are having a generator tech come onsite to determine why they failed. We received an email from David Gleaner, the COO of Evocative, that a deeper investigation is being conducted. At Carolina Digital Phone, we did not bring our voice and messaging services in Dallas, TX, back onto the geo-redundant cluster until a diagnosis could be completed to ensure that the data stored on these servers was complete and without corruption. After completing this test, we restored the connection to our two other data centers, bringing the full geo-redundancy back into operation. No calls were lost, nor did customers experience any outages.