ETHERFAX incident

Investigating Reports of Certain Calls Not Reaching etherFAX

Minor Resolved View vendor source →

ETHERFAX experienced a minor incident on June 26, 2024 affecting U.S. Inbound DID, lasting 1d 1h. The incident has been resolved; the full update timeline is below.

Started
Jun 26, 2024, 07:37 PM UTC
Resolved
Jun 27, 2024, 09:03 PM UTC
Duration
1d 1h
Detected by Pingoru
Jun 26, 2024, 07:37 PM UTC

Affected components

U.S. Inbound DID

Update timeline

  1. investigating Jun 26, 2024, 07:37 PM UTC

    Engineers are currently engaged with one of our underlying carriers to determine why calls to some US DIDs are not reaching the etherFAX network.

  2. investigating Jun 26, 2024, 08:18 PM UTC

    etherFAX engineers remain in contact with the carrier NOC and engineers while the issue continues to be investigated.

  3. investigating Jun 26, 2024, 10:31 PM UTC

    The carrier ticket has been escalated to tier II and the engineering group within the affected underlying carrier. The etherFAX team remains engaged with the carrier as we await further updates.

  4. investigating Jun 27, 2024, 02:33 PM UTC

    The etherFAX team has observed some restoration. We are still engaged and working with the carrier to ensure the issue is fully resolved.

  5. investigating Jun 27, 2024, 05:58 PM UTC

    Carrier has identified possible cause and remedied. We are continuing to test and monitor. More details to follow.

  6. resolved Jun 27, 2024, 09:03 PM UTC

    We are resolving this incident based on our continued monitoring along with confirmation from our carrier that the failed equipment was replaced. We expect to have a full RFO from them within 72 business hours.

  7. postmortem Jul 02, 2024, 06:55 PM UTC

    etherFAX has received RCA for this incident from carrier. **Incident Start Date:** 6/26/24 18:31 GMT **Incident End Date:** 6/27/24/17:41 GMT **Cause**: * 2 card slots on carrier switch backplane were in an unexpected state. Bad states caused the intermittent call failures. **Resolution:** * Affected card slots were reset. Calls began completing normally. **Planned Actions:** * Carrier has added additional logging on this hardware component to scrape, log, and alert on the trouble code. * Carrier NOC has implemented additional communication pathways to their engineering team. * Carrier Executive Team participated in a call with etherFAX leadership to discuss the outage event and lessons learned.