StrongDM incident

SDK: Service Degradation

Major Resolved View vendor source →

StrongDM experienced a major incident on September 30, 2021 affecting API, lasting 8h 15m. The incident has been resolved; the full update timeline is below.

Started
Sep 30, 2021, 03:48 PM UTC
Resolved
Oct 01, 2021, 12:04 AM UTC
Duration
8h 15m
Detected by Pingoru
Sep 30, 2021, 03:48 PM UTC

Affected components

API

Update timeline

  1. investigating Sep 30, 2021, 03:48 PM UTC

    strongDM has identified a known issue that seems to be related to Let's Encrypt certificate expiration. strongDM is actively remediating.

  2. identified Sep 30, 2021, 05:31 PM UTC

    Remediation steps are being initiated which we expect to result in a service outage for a few minutes.

  3. identified Sep 30, 2021, 05:49 PM UTC

    Further remediation steps are being investigated

  4. identified Sep 30, 2021, 06:05 PM UTC

    Remediation steps to push a new certificate CA to our production servers. This may cause a service outage of a few minutes.

  5. identified Sep 30, 2021, 06:57 PM UTC

    strongDM has identified the source of the SDK degradation as an expired certificate issued by Let's Encrypt. strongDM has released remediating steps to update the DNS records and issued requests to DNS providers to refresh relevant entries. strongDM is awaiting DNS propagation. If any customer wishes to accelerate their ability to use our API before DNS propagates, please reach out to [email protected] for a workaround.

  6. resolved Oct 01, 2021, 12:04 AM UTC

    DNS has successfully propagated and all systems are operational.