DAT Rateview credentials not used

Update timeline

1. resolved Nov 01, 2022, 11:58 PM UTC

   Issue Summary We had a change in our system that caused customer DAT Rateview credentials to be incorrectly marked as invalid, and therefore not used when fetching updated Rateview data. Timeline We first noticed this issue when one of our automated alerts notified our support team that a majority of customer Rateview credentials were marked as invalid. This happened around 7am EST on 11/1/2022 Support took a dual pronged approach, alerting both Parade's internal team, and also the DAT team to identify if these Rateview credentials were actually not working. After internal investigation, we identified that a deployment the night before on 10/31/2022 had added a check to mark DAT Rateview credentials as invalid. Root Cause This additional check deployed the night before, had caused most of our credentials to be identified as "invalid" and therefore not usable for the integration with DAT Rateview. The end result was that this check was too broad, and was preventing valid Rateview credentials from being used. Resolution and recovery Parade team reverted the breaking change around 10:09 AM PST on 11/1/2022. This resolved the issue, and we also re-enabled all Rateview creds around the same time. Any new loads or updated loads after that time would have had their Rateview rates automatically re-fetched. All Rateview rates on prior available loads within Parade were refreshed as part of the daily sync around 4pm PST. No customer data was lost during this time period. Corrective and Preventative Measures Parade's internal alerting mechanisms worked as intended in this scenario. Our internal support and engineering teams were notified promptly about the Rateview inconsistencies. We will continue monitoring our internal alerts around Rateview going forward, and make this a standard part of our customer health checks. We have put in a ticket for the engineering team to implement a longer term solution to identify invalid Rateview credentials, as we still do not believe we have a great way to identify these post initial onboarding. It is a common problem for customers to change their credentials or lose access to subscriptions. The Parade team will continue working with DAT to identify these bad credentials going forward so that customers do not experience any additional interruptions to their Parade - Rateview integration.