Intermittent time-outs towards OSR provider

Update timeline

1. investigating Mar 01, 2022, 07:20 AM UTC

   Customers may experience intermittent errors accessing the CSP, DCA and MNO Web Portals. Additionally, customers may experience intermittent time-out errors when trying to access TCR API endpoints. This is currently impacting Brand and Campaign registration. The issue started at approximately 12:40 PM EST. It is currently under investigation.

2. identified Mar 01, 2022, 07:21 AM UTC

   Customers may experience intermittent time-out errors while trying to register Campaigns via the TCR Web Portal or API. This issue is due to time-outs towards our OSR partner. The incident has been reported to our OSR partner. Additionally, TCR engineers are working to mitigate the issue while our OSR partner works towards a resolution. We recommend our customers to enable their retry functionality if a time-out error is encountered while trying to register a Campaign.

3. identified Mar 01, 2022, 07:22 AM UTC

   We have a fix for this time-out issue and now are testing it to push to production soon.

4. monitoring Mar 01, 2022, 01:27 PM UTC

   We have implemented the fix, tested it and pushed to production. Will actively monitor production results.

5. resolved Mar 02, 2022, 01:40 AM UTC

   We have updated our system to mitigate the issue when there is a downstream delay in our partner's network. Any time-out events experienced yesterday should now be resolved and we recommend enabling your retry mechanism to complete Campaign updates/registrations. Our team continues to monitor for any recurrence.