Campaign Registry Outage History

There is a current issue with Okta support for SMS as a method for Multi-Factor Authentication (MFA). While we investigate further, in order to allow all customers using SMS as their MFA method to login to the portals, we have reset MFA for all users. There is no outage, but you will need to re-enable an MFA method on your next login to any of the portals. We have added support for email as an MFA method in order to support those that do not have an authentication application (like Google Authenticator or Okta Authenticator). We apologize for the inconvenience.

The intermittent high load issue that was causing some queries to time out should be resolved now. We will continue to monitor, but will close out this incident for now. If you are still seeing issues, please open a support ticket ([email protected])

Resolved - One of the 3rd party tools the CSP portal uses for access to external links was impacted this morning by the Cloudlflare outage. This began about 7:30am and was resolved around 9:30 am EST external to TCR. The portal itself remained fully functional, but links to externally hosted data (links to the Terms and Conditions, the Knowledge Base, etc) were not functioning and you may have seen an error upon first logging in. The issue is now fully resolved and all external links are functioning as well. If you do see any issues, please open a support ticket by emailing [email protected] NOTE: There was no impact to the API at any time.

Immediately after our scheduled release this morning, we encountered an increasing backlog of queries related to campaign sharing. This caused significant slowdowns, up to and including timing out, for the CSP Portal, CSP API, DCA Portal and DCA API. We have identified the issue and hotfix has been released. All issues appear to have been resolved. During the outage nnSR component was up the whole time. nnSR downtime was false positive. The nnSR component status was also corrected to operational.

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.