SSL certificate issue - unable to get local issuer certificate

Update timeline

1. resolved Jun 03, 2025, 03:35 AM UTC

   Type: Incident Duration: 16 hours Affected Components: Osaka, Oregon, Sao Paulo, Canada, Tokyo, London, Ohio, N California, Seoul, Frankfurt, Cape Town, Ireland, N Virginia, Sydney, , Stockholm, Singapore, Paris, Milan, Mumbai, Cluster Regions → Jun 3, 10:20:00 GMT+0 - Identified - We started getting sporadic reports of a few users running into SSL certificate issues when connecting to their Typesense Cloud clusters. On closer investigation, we found that they were trying to connect to Typesense from older operating systems or older runtimes that bundle their own root CAs (like Node.js), which hadn't updated their Root CA store recently. Updating their OS / runtime CA certificates, to the latest version fixed the issue for them. Our SSL provider started using a new Root CA certificate to sign our certificate this year, and this new root CA certificate wasn't available on older runtimes. So manually updating the root CA store on users' computers fixed the issue, so that the SSL cert presented by Typesense Cloud was able to be validated successfully. Jun 3, 18:45:00 GMT+0 - Monitoring - As the volume of reports started increasing, to prevent more users on older OS / runtimes from running into this issue, we decided to roll back to the previous SSL certificate temporarily. We've also canceled the scheduled roll out of this new SSL certificate to existing clusters. If you're running into SSL cert issues, and if you provisioned a cluster or made config changes during the time frame of this incident, you want to do a \[rolling restart\](https://cloud-help-center.typesense.org/article/33-restart-typesense-cloud-cluster) to refresh your SSL certificate. Alternatively, the certs will also be automatically rolled back in the next few hours. We're now going to be working with our SSL cert provider to identify how to best roll out this new Root CA without affecting older runtimes. This issue only affected: 1) Clusters that were provisioned during the duration of this incident 2) Clusters whose configuration (RAM, CPU, Typesense Version, etc) was changed during the duration of this incident. Existing clusters that were already running prior to the duration of this incident were unaffected. Jun 3, 19:34:53 GMT+0 - Resolved - We're now going to be working with our SSL cert provider to identify how to best roll out this new Root CA without affecting older runtimes. This issue only affected: 1) Clusters that were provisioned during the duration of this incident 2) Clusters whose configuration (RAM, CPU, Typesense Version, etc) was changed during the duration of this incident. Existing clusters that were already running prior to the duration of this incident were unaffected. Jun 3, 03:35:00 GMT+0 - Investigating - We rotate our SSL certificates for Typesense Cloud annually as part of our standard operating procedure. We tested and deployed a new SSL cert for _new_ nodes on Typesense Cloud (provisioned either during new cluster creation or via config changes) at 3:30AM UTC on Jun 2, 2025. We also scheduled the new cert to be pushed live to existing nodes at 12:00AM UTC on Jun 3, 2025.