SSL Certificate Changes to Public Platform APIs

Update timeline

1. investigating Nov 11, 2024, 03:13 PM UTC

   Around 11/11 8:00 UTC our SSL certificate chain for our driveaxleapp.com domain was updated. This update may have caused issues connecting to our public facing platform APIs. We are investigating the issue now.

2. identified Nov 11, 2024, 03:39 PM UTC

   The recently issued certificate no longer contains Starfield C2 in it's certificate, this issue will manifest to any services that rely on Starfield C2 to be contained in the certificate chain. If your integration or systems do not rely on Starfield C2 to be contained in the certificate chain, or otherwise, you are not seeing any issues connected to our API endpoints, then you can ignore this message. If you are experiencing any issues connecting or making API calls to the Eleos Platform, please reach out to our support team at [email protected]

3. monitoring Nov 11, 2024, 04:15 PM UTC

   For anyone having any SSL certificate issues when connecting to the Eleos APIs, please ensure that you add Amazon's CAs to your trust store. For more information, please see Amazon's guide on this: https://aws.amazon.com/blogs/security/acm-will-no-longer-cross-sign-certificates-with-starfield-class-2-starting-august-2024/ Again, this issue is limited to those who rely on Starfield C2 being contained within the certificate chain, if your systems do not rely on this, or you aren't experiencing any issues connecting to the Eleos APIs, please disregard. For any questions, please reach out to [email protected]. We apologize for this inconvenience.

4. monitoring Nov 11, 2024, 05:28 PM UTC

   For those still having any SSL certificate issues when connecting to the Eleos APIs, please ensure that you add Amazon's CAs to your trust store. For more information, please see Amazon's guide on this: https://aws.amazon.com/blogs/security/acm-will-no-longer-cross-sign-certificates-with-starfield-class-2-starting-august-2024/ For any questions, please reach out to [email protected].

5. resolved Nov 11, 2024, 07:21 PM UTC

   For those still having any SSL certificate issues when connecting to the Eleos APIs, please ensure that you add Amazon's CAs to your trust store. For more information, please see Amazon's guide on this: https://aws.amazon.com/blogs/security/acm-will-no-longer-cross-sign-certificates-with-starfield-class-2-starting-august-2024/ As a follow-up, we have confirmed that the updated certificate only impacted customers' backoffice systems contacting our public API, and that supported mobile app versions were able to connect to the Eleos API and were not directly affected. Impacted customer systems may have caused issues with drivers' mobile apps. For example, if your messaging web service was affected, your drivers would not have received messages sent to them until your service could contact the Eleos API. However, messages sent by drivers were delivered successfully.

6. postmortem Nov 26, 2024, 04:18 PM UTC

   On 2024-11-11 at 8:12 UTC, as part of standard TLS certificate management procedures, our systems automatically renewed the certificate used by [platform.driveaxleapp.com](http://platform.driveaxleapp.com) ahead of its upcoming expiration. Some customer integrations did not recognize the validity of the new certificate, and were thus unable to initiate secure connections to the Eleos Platform API until they were updated to do so. ## Impact Affected customers’ integrations were able to connect to the Eleos Platform API, but aborted the connection before making a request because they were unable to determine the validity of the TLS certificate presented by the server. Primarily, this prevented messages from being delivered to drivers via the `PUT /api/v1/messages/{handle}` API endpoint. Depending on the integration, other API calls may have also been affected, including: * Load updates and deletes via `/api/v1/users/{username}/`... endpoints * User refreshes via `POST /api/v1/users/{username}/updates` * API-based app management, such as form and screens * API-based platform data management, such as trip plan fetching Mobile app communication with Eleos backend systems was not affected directly. The issues with the noted customer systems indirectly degraded API-driven functionality, such as backoffice-to-driver message delivery. App Manager and Document Hub were not affected. Customers with an integration using an up-to-date root store were not impacted. ## Background HTTPS APIs such as the Eleos Platform API rely on the web’s Public Key Infrastructure \(PKI\) to allow client applications to verify the identity of the server and prevent machine-in-the-middle attacks. As part of PKI, clients such as web browsers, HTTP libraries, and language runtimes provide a “root store” of pre-trusted certificates controlled by audited third parties, called certificate authorities \(CAs\), who cryptographically certify the issuance of certificates for individual websites. These pre-trusted certificates form the “root of trust” for verifying server identities. For a successful connection, the server must present to the client a certificate that is transitively signed by one or more of these roots, and the client must have at least one of those same roots pre-loaded in its root store. If the server presents a certificate signed by a CA that is not present in the client’s root store, the connection will fail. From time to time, new CAs pass the necessary audits and are included in major root stores, and some CAs go out of business or fail to comply with necessary policies and are proactively removed, since they can no longer be trusted to properly verify certificates. Finally, root certificates are only valid for a finite period of time. For these reasons, root stores are not static, and must receive periodic updates to ensure continued interoperability with the greater web, including the Eleos Platform APIs. In this instance, the previous certificate for [platform.driveaxleapp.com](http://platform.driveaxleapp.com) was rooted with the Starfield Class 2 Certification Authority certificate. Because of a new policy in Mozilla and Chromium’s root store programs that limits the lifetime of any given root certificate to 15 years, this CA will no longer be trusted by these major root stores in April 2025. To ensure no certificates remain in use past that date, our certificate issuer is transitioning away from it proactively. Both the new and expiring certificates included two Amazon-managed root CAs as part of their trust chains, Starfield Services Root Certificate Authority - G2 and Amazon Root CA 1. These root CAs date to 2011 and 2017, respectively, and have been widely included in the root stores used by browsers and operating systems since that time. Because Amazon Root CA 1 is cross-signed by ​​Starfield Services Root Certificate Authority - G2, trusting either root CA was sufficient to be unaffected by the change. However, affected systems relied on the older Starfield Class 2 CA to validate the [platform.driveaxleapp.com](http://platform.driveaxleapp.com) certificate and did not include either of the new CAs in their root store. As a result, those systems were unable to confirm the validity of the new certificate and rejected the attempted connections to the Eleos Platform API. To avoid Eleos Platform outages caused by expired certificates, certificate renewal is a fully automated operation, and certificates are designed to expire relatively frequently to ensure these mechanisms are exercised regularly. These certificates can change at any time, even ahead of the end of their validity period. We use multiple, distinct monitoring systems to ensure our APIs present widely-trusted and valid certificate chains at all times, and both automation and monitoring functioned as expected during this incident. Our commitment is to continue to use certificates issued by established, trusted CAs that are present in the major \(CCADB, Chromium, Mozilla, Microsoft, Apple, Java\) root stores. To ensure API clients are able to successfully validate the authenticity of Eleos APIs, we recommend root stores be kept up to date via the mechanisms available through operating system and language runtime vendors. We do not recommend “pinning” or manually trusting observed certificates by adding them to a custom root store.