Divio incident

SSL issues following the change of Let's Encrypt root certificate

Major Resolved View vendor source →

Divio experienced a major incident on September 30, 2021 affecting Divio Single Sign-On and Divio Single Sign-On and 1 more component, lasting 21h. The incident has been resolved; the full update timeline is below.

Started
Sep 30, 2021, 03:24 PM UTC
Resolved
Oct 01, 2021, 12:24 PM UTC
Duration
21h
Detected by Pingoru
Sep 30, 2021, 03:24 PM UTC

Affected components

Divio Single Sign-OnDivio Single Sign-OnDivio Single Sign-OnDivio Single Sign-On

Update timeline

  1. investigating Sep 30, 2021, 03:24 PM UTC

    We and some customers experience SSL issues following the change of Let's Encrypt root certificate. This can for example happen if your application tries to connect to external web services or it is using divio SSO. Depending on the base image you are using, the root certificate bundle might be too old and relies on a root certificate for Let's Encrypt that expired today: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/. If you are using python, you might have to upgrade the version of requests and/or certifi in your application in order for it to recognize the Let's Encrypt certificates used by control.divio.com and other domains again. Our GIT and SSH services are currently also impacted by this and we are working on a resolution.

  2. monitoring Sep 30, 2021, 04:58 PM UTC

    We have changed our own control.divio.com certificate to be more compatible with older root certificate bundles. This change should resolve issues with Divio SSO. Despite our change, we strongly advice our clients to also update root certificate bundles and related packages of their applications to a newer version as soon as possible.

  3. resolved Oct 01, 2021, 12:24 PM UTC

    This incident will be considered as resolved. The new certificate is working as expected, git and SSH services are restored. Customers might still experience issues with other providers and we continue to advice our clients to also update root certificate bundles and related packages of their applications to a newer version as soon as possible.