Invoice generation, sending, and payment processing is delayed for some accounts

Update timeline

1. investigating May 30, 2020, 06:09 PM UTC

   Some ChargeOver accounts are experiencing delayed invoice generation, invoice sending, and automated payment processing. We are investigating.

2. monitoring May 30, 2020, 07:23 PM UTC

   We have resolved the issue, and services are being restored. Invoices are generating, emails are sending, and payments are processing as normal now. Some ChargeOver accounts may see delayed generation of invoices while things catch back up. A further update will follow.

3. monitoring May 30, 2020, 09:29 PM UTC

   We continue to monitor things, as things catch up -- invoices are generating, emails are sending, and payments are processing.

4. resolved May 31, 2020, 12:14 AM UTC

   This issue has been resolved. Postmortem to follow.

5. postmortem Jun 03, 2020, 03:55 PM UTC

   **What Happened / Customer Impact** Our certificate authority issued ChargeOver a TLS/SSL certificate with a CA chain certificate which expired before our actual TLS/SSL certificate did. This caused automatic invoice generation and payment processing to be delayed for some customers. **Technical Details** ChargeOver secures internal and external connections with TLS/SSL certificates. The company ChargeOver purchases certificates from issued us an SSL/TLS certificate with a chain file which expired before our actual certificate. So even though our SSL/TLS certificate was valid and _not_ expired, a certificate in the chain needed to validate the certificate expired on May 30th. Most web browsers were unaffected, so access to the [ChargeOver.com](http://ChargeOver.com) website and app were unaffected. However, common libraries and tools like cURL and wget began rejecting connections due to the expired chain certificate. ChargeOver's scheduled invoice generation and scheduled payment processes depend on the cURL library, and so the system was unable to trigger invoice generation and payment processing for some accounts scheduled to generate invoices after the certificate had expired. Our monitoring picked up the issue, and alerted our engineering team. ChargeOver then removed the expired certificate from the chain, and invoices and payments began processing normally again. This caused a delay of invoice/payment processing of between 30 minutes and 4 hours for some ChargeOver accounts. **Ongoing Efforts** We are working on some additional validation to ensure that our CA cannot issue us certificates with a certificate chain that expires prior to the certificate we are being issued.