Campaign Registry incident
Intermittent time-outs towards OSR provider
Campaign Registry experienced a major incident on February 28, 2022 affecting csp-api and dca-api and 1 more component, lasting 18h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 01, 2022, 07:20 AM UTC
Customers may experience intermittent errors accessing the CSP, DCA and MNO Web Portals. Additionally, customers may experience intermittent time-out errors when trying to access TCR API endpoints. This is currently impacting Brand and Campaign registration. The issue started at approximately 12:40 PM EST. It is currently under investigation.
- identified Mar 01, 2022, 07:21 AM UTC
Customers may experience intermittent time-out errors while trying to register Campaigns via the TCR Web Portal or API. This issue is due to time-outs towards our OSR partner. The incident has been reported to our OSR partner. Additionally, TCR engineers are working to mitigate the issue while our OSR partner works towards a resolution. We recommend our customers to enable their retry functionality if a time-out error is encountered while trying to register a Campaign.
- identified Mar 01, 2022, 07:22 AM UTC
We have a fix for this time-out issue and now are testing it to push to production soon.
- monitoring Mar 01, 2022, 01:27 PM UTC
We have implemented the fix, tested it and pushed to production. Will actively monitor production results.
- resolved Mar 02, 2022, 01:40 AM UTC
We have updated our system to mitigate the issue when there is a downstream delay in our partner's network. Any time-out events experienced yesterday should now be resolved and we recommend enabling your retry mechanism to complete Campaign updates/registrations. Our team continues to monitor for any recurrence.