ETHERFAX experienced a major incident on March 6, 2025 affecting U.S. REST Services and Canada REST Services and 1 more component, lasting 13d 17h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 06, 2025, 01:16 AM UTC
ETHERFAX engineers are investigating an ISP level routing issue impacting connectivit to the API. Changes have been initiated to route around the impacted ISP.
- identified Mar 06, 2025, 01:42 AM UTC
Our primary ISP has reported a widespread issue. We have completed DNS updates to route API traffic around the affected provider. Engineers are continuing to monitor services and failover processes.
- identified Mar 06, 2025, 03:26 AM UTC
Most services have restored fully through alternate paths. Engineers continue to work with our ISP and monitor performance.
- monitoring Mar 06, 2025, 05:45 AM UTC
All services are operating as expected. For any applications that remain unable to access our API or endpoints, ensure that outbound firewall rules permit this access. DNS changes made during failover events mean that hard-coded IP address rules or DNS entries may prevent normal operation. Engineers will continue to monitor performance of the network into tomorrow.
- monitoring Mar 06, 2025, 04:07 PM UTC
Engineers are beginning to move some services back to primary ISP. An issue was noted with DirectFax Email Services rejecting emails due to a configuration error with our domain validation check. This was just resolved.
- resolved Mar 19, 2025, 06:24 PM UTC
The affected ISP has issued their RFO and advised a hardware card failure on their network backbone. An ISP field engineer dispatch was required to move traffic off. A new card was shipped to site and was replaced by the field engineer. Note, ETHERFAX kept the incident open although in a monitoring state until the repair was completed by the ISP.