Parade experienced a major incident on November 1, 2022, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Nov 01, 2022, 11:48 PM UTC
Issue Summary Carriers were unable to log in to Parade for Carriers due to a bad deployment of new database credentials. This issue did not affect any broker related portals. Carriers were still able to log in to Parade when coming in from an email link. The only login portal affected was the one hosted on carriers.parade.ai Timeline This issue was first identified at 7AM EST on 10/31/2022. After investigation by the Parade team, we restored the database connection around 10:24 AM PST on 10/31/2022. Root Cause The issue was caused by a bad deployment that unhooked our Parade for Carriers app from its supporting database. This caused any APIs that relied on the database to fail for the carrier app. Resolution and recovery Resolution was achieved by redeploying our Parade for Carriers backend APIs with a new set of database connection credentials. Recovery to the platform was achieved soon after redeploy at 10:24 AM PST. No customer data loss occurred, but it did take our platform ~45 minutes to replay all events going to the carrier portal. Corrective and Preventative Measures We have added additional monitoring and alerting for the Parade for Carriers app specifically. We have also provisioned new database secrets for our deployments so that any future deployments will no longer have a database connection mixup. This also adds a side effect of bringing in additional security to our platform.