TSG Global experienced a minor incident on April 6, 2023 affecting Local Inbound/Outbound SMS, lasting 20d 19h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 05, 2023, 06:47 PM UTC
We have reports from customers that duplicate text messages are being received. We are aware of the issue and are working with our aggregator to resolve the issue/implement a fix. It is intermittent, and will have an update soon.
- investigating Apr 05, 2023, 08:25 PM UTC
Firstly, we apologize for the inconvenience. We are continuing to work with our aggregator to fix this issue via Zoom. We have been in constant contact with their technical team, and the root issue is stemming from our aggregator's side - not TSG Global. We will provide a full postmortem once the issue is resolved. If you have any questions, please reach out to [email protected]
- monitoring Apr 05, 2023, 11:36 PM UTC
A fix has been implemented and we are monitoring throughout the evening for any new issues. We will resolve this incident in the AM on 4/6/2023 if no further issues are encountered. Again, we appreciate your patience as we worked with our aggregator to resolve the issue, and will provide a full postmortem once we collect all the necessary details.
- identified Apr 06, 2023, 06:03 PM UTC
Firstly, we apologize for the continued inconvenience. We continue working with our aggregator to fix this issue. We have been in constant contact with their technical team, and the root issue stems from our aggregator's side - not TSG Global. We will provide a full postmortem once the issue is resolved. If you have any questions, please reach out to [email protected]
- identified Apr 06, 2023, 08:43 PM UTC
1:30 PM PST update: we have been on calls continually with our aggregator today attempting to resolve this ongoing issue. We will continue working through the night and the early AM to get this issue resolved for you. Unfortunately, we don't have an explicit ETA at this time. We can share that issue is transient in nature and is randomly affecting a subset of our overall traffic for all customers. We are performing some deep network analysis (PCAPs, etc.) and queue analysis in tandem with our aggregator to get this fixed as soon as possible. We will provide another update soon.
- identified Apr 07, 2023, 05:59 PM UTC
10:53 AM PST Update: we are still working with our aggregator to work on correcting this issue, and have been actively on a call with them all morning. We are migrating traffic between binds and performing other tests to find the root cause and get it fixed. Unfortunately, due to the complexity of this issue, implementing a temporary fix at the software layer to prevent duplicate messages is a last-resort option. We will provide an update again later this morning.
- identified Apr 07, 2023, 07:12 PM UTC
12:10 PM PST Update: there was a temporary outbound message delay while we troubleshooted the issue with our aggregator. All messages were queued during that period, and the queue has now drained. We are now going to be disabling delivery receipts (DLRs) to continue testing as we are seeing positive momentum towards getting this resolved. Please be advised that you may not receive DLRs for a period of time while we continue working to fix this issue.
- monitoring Apr 07, 2023, 09:32 PM UTC
The aggregate gateway has implemented a solution and we will be closely monitoring its performance with them over the next 72 hours. During this time, their system is processing a backlog of inbound messages, and you may experience some duplicate message deliveries. Please note that new incoming messages may be temporarily delayed as we clear the queue. Delivery Receipt (DLR) reports are currently suspended, and we will provide an update once they resume. We appreciate your patience and understanding.
- monitoring Apr 08, 2023, 02:18 PM UTC
We are delighted to announce that the aggregator's patch has been effective, with no errors reported thus far. As of 14:00 UTC (10:00 EST), instances of duplicate messages have significantly decreased, although the backlog queue is still being processed. New outgoing messages might experience temporary delays while we continue clearing the queue. Please be aware that Delivery Receipt (DLR) reports remain suspended for now, and we will share an update once they are back online. Thank you for your patience and understanding during this time.
- monitoring Apr 09, 2023, 02:03 AM UTC
We are pleased to report that the backlog queue has been processed, and there are no longer any delays with inbound messages. Delivery Receipt (DLR) reports remain suspended, and we'll provide further updates as the situation evolves. We appreciate your continued patience and understanding.
- monitoring Apr 09, 2023, 02:07 PM UTC
Our team continues to closely monitor normal service levels for SMS messages following the clearing of the backlog queue. Please note that Delivery Receipt (DLR) reports remain suspended at this time. We appreciate your ongoing patience and will provide another update in 12 hours.
- monitoring Apr 10, 2023, 02:19 AM UTC
Service levels for messaging continue to operate at expected levels. Please note that Delivery Receipt (DLR) reports remain suspended at this time. We appreciate your ongoing patience and will provide another update in 12 hours.
- monitoring Apr 10, 2023, 02:22 PM UTC
We're pleased to confirm that the situation remains stable; messaging continues to operate at expected levels. Delivery Receipt (DLR) reports, however, continue to be suspended. We'll keep monitoring the system to ensure optimal performance and provide updates on DLR reports as they become available. Thank you for your patience and understanding throughout this process.
- monitoring Apr 11, 2023, 01:08 PM UTC
Messaging continues to operate at expected levels. Delivery Receipt (DLR) reports, however, continue to be suspended. We'll keep monitoring the system to ensure optimal performance and provide updates on DLR reports as they become available. Thank you for your patience and understanding throughout this process.
- monitoring Apr 13, 2023, 02:08 PM UTC
Messaging continues to operate at expected levels. Delivery Receipt (DLR) reports, however, continue to be suspended. We continue to work with our aggregate to resume DLR delivery, and we will provide updates on DLR reports as they become available. Thank you for your patience and understanding throughout this process.
- monitoring Apr 20, 2023, 12:02 PM UTC
We are excited to share an important update regarding Delivery Receipt (DLR) reports. As promised in our last communication, we have continuously worked with our aggregate to resume DLR delivery. We are now pleased to announce that we have made significant progress and are gradually enabling DLR delivery requests for clients who request them. This process will be incremental as we closely monitor the status to ensure optimal performance and reliability. Our team is dedicated to maintaining the high quality of our messaging services, and we appreciate your patience and understanding throughout this transition. Stay tuned for more updates as we work towards fully restoring DLR delivery for all clients. Should you have any questions or concerns, please don't hesitate to contact our support team. Thank you for your continued trust in our services!
- resolved Apr 27, 2023, 03:52 PM UTC
This incident has been resolved. DLRs are available to all clients who have been requesting them.