TPG Telecom experienced a minor incident on July 22, 2024 affecting Web Portal, lasting 1d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 22, 2024, 12:16 AM UTC
We are currently experiencing delays with Outbound SMS and Reporting. Our teams have been alerted and are actively working to identify and resolve the issue. We recommend that customers refrain from resending campaigns and allow time for message processing to catch up. Impact: - Outbound message sending is delayed. - Delivery receipts are delayed. - Message reporting is delayed. Thank you for your patience as we work to resolve these issues.
- identified Jul 22, 2024, 12:29 AM UTC
The issue has been identified and a fix is being implemented.
- identified Jul 22, 2024, 02:50 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Jul 22, 2024, 03:12 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jul 22, 2024, 03:41 AM UTC
We are continuing to monitor for any further issues.
- identified Jul 22, 2024, 04:28 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 22, 2024, 08:30 AM UTC
A fix has been implemented and we are monitoring the results.
- identified Jul 22, 2024, 10:53 PM UTC
The issue has been identified and a fix is being implemented. Outbound sending is not impacted. Reporting services are 1.5 hours delayed. Impact: - Inbound message receiving is delayed for inbox - Delivery receipts are delayed. - Message reporting is delayed. We recommend that customers refrain from resending campaigns and allow time for message processing to catch up.
- identified Jul 23, 2024, 12:15 AM UTC
We are continuing to work on a fix for this issue. Outbound sending is not impacted. Reporting services are 1.5 hours delayed. Impact: - Inbound message receiving is delayed for inbox - Delivery receipts are delayed. - Message reporting is delayed. We recommend that customers refrain from resending campaigns and allow time for message processing to catch up.
- identified Jul 23, 2024, 01:26 AM UTC
We are continuing to work on a fix for this issue. Outbound sending is not impacted. Reporting services are 1.2 hours delayed. Impact: - Inbound message receiving is delayed for inbox - Delivery receipts are delayed. - Message reporting is delayed. We recommend that customers refrain from resending campaigns and allow time for message processing to catch up.
- identified Jul 23, 2024, 02:39 AM UTC
We are continuing to work on a fix for this issue. Outbound sending is not impacted. Reporting services are 1.2 hours delayed. Impact: - Inbound message receiving is delayed for inbox - Delivery receipts are delayed. - Message reporting is delayed. We recommend that customers refrain from resending campaigns and allow time for message processing to catch up.
- identified Jul 23, 2024, 04:44 AM UTC
We have identified the root cause of this issue and are working on remediation activities. Outbound messages are being sent as normal, however they may not show as being sent or delivered within the web portal. There are delays to updating the sent messages page, campaign reports and inbox. Message delivery reporting and Inbox notifications services are still experiencing delays between 15 and 90 minutes. Recommendations: - Customers are advised not to resend campaigns currently. - Please allow time for message delivery reporting and inbox notification processing to catch up.
- monitoring Jul 23, 2024, 05:28 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 23, 2024, 06:57 AM UTC
This incident has been resolved.