CallTrackingMetrics incident

US Toll Free and Subset of US Long Code Numbers Experiencing Inbound and Outbound SMS and MMS Delivery Delays to/from US and Canada Networks

Notice Resolved View vendor source →

CallTrackingMetrics experienced a notice incident on January 14, 2022 affecting Plivo Longcode SMS, USA & Canada and Plivo Toll-free SMS, USA & Canada and 1 more component, lasting 1d 4h. The incident has been resolved; the full update timeline is below.

Started
Jan 14, 2022, 04:17 PM UTC
Resolved
Jan 15, 2022, 08:33 PM UTC
Duration
1d 4h
Detected by Pingoru
Jan 14, 2022, 04:17 PM UTC

Affected components

Plivo Longcode SMS, USA & CanadaPlivo Toll-free SMS, USA & CanadaPlivo MMS, USA & CanadaTwilio MMS Toll-FreeTwilio MMS Long Code, North AmericaTwilio SMS Long Code, North America

Update timeline

  1. investigating Jan 14, 2022, 03:56 PM UTC

    We are continuing our investigation and have discovered inbound traffic to Twilio Toll-Free numbers from US and Canada networks is also affected and experiencing delivery delays. In addition, we have discovered a small portion of US long code numbers are affected and also experiencing SMS and MMS delivery delays for inbound and outbound traffic to/from US and Canada networks. Our team continues working with our carrier partner to resolve the issue. We will provide another update in 2 hours or as soon as more information becomes available.

  2. investigating Jan 14, 2022, 03:59 PM UTC

    We are continuing to investigate this issue.

  3. identified Jan 14, 2022, 04:00 PM UTC

    The issue has been identified and a fix is being implemented.

  4. monitoring Jan 14, 2022, 04:17 PM UTC

    We are continuing our investigation and have discovered inbound traffic to Twilio Toll-Free numbers from US and Canada networks is also affected and experiencing delivery delays.

  5. identified Jan 14, 2022, 04:18 PM UTC

    The issue has been identified and a fix is being implemented.

  6. identified Jan 14, 2022, 04:53 PM UTC

    We are now observing some recovery in which some SMS and MMS to/from US and Canada networks from US Toll-Free and US long codes are being delivered. Our engineers have identified the issue and implemented a workaround for older messages. However, message DLRs will continue to be delayed even while messages are getting delivered. Our engineers are continuing to work with our carrier partner to resolve the issue. We will provide another update in 4 hours or as soon as more information becomes available. (from upstream)

  7. monitoring Jan 14, 2022, 05:59 PM UTC

    A fix has been implemented and we are monitoring the results.

  8. identified Jan 14, 2022, 08:56 PM UTC

    While we've observed noticeable recovery after the implementation of the workaround, we continue to see intermittent delivery delays sending and receiving Toll Free SMS and MMS messages in US and Canada, along with a small subset of T-mobile traffic on US long codes. The issue is also causing delivery receipts to be delayed, with delivery status stuck at "Sent". Our engineering team is continuing to work with our carrier partner to fully resolve the issue. We will provide another update in 1 hour or as soon as more information becomes available.

  9. resolved Jan 15, 2022, 08:33 PM UTC

    This incident has been resolved.