Rebrandly incident

HTTPS redirection issues: revoked certificate

Major Resolved View vendor source →

Rebrandly experienced a major incident on October 13, 2021 affecting Branded Links redirect (Domain Names registered via Rebrandly) and Branded Links redirect (Domain Names managed outside Rebrandly) and 1 more component, lasting 22h 33m. The incident has been resolved; the full update timeline is below.

Started
Oct 13, 2021, 09:15 PM UTC
Resolved
Oct 14, 2021, 07:48 PM UTC
Duration
22h 33m
Detected by Pingoru
Oct 13, 2021, 09:15 PM UTC

Affected components

Branded Links redirect (Domain Names registered via Rebrandly)Branded Links redirect (Domain Names managed outside Rebrandly)Generic short URLs redirect

Update timeline

  1. identified Oct 13, 2021, 09:15 PM UTC

    We are experiencing downtime in HTTPS redirection due to an abrupt revoke of a number of SSL certificates which were generated by Let's Encrypt Certificate Authority. We are in the process of re-generating SSL certificates to restore redirection. Link redirection for customers with Custom SSL certificates is not affected.

  2. identified Oct 13, 2021, 10:25 PM UTC

    A patch is on the go, as we re-issue the revoked SSL certificates for affected domain names. We are in touch with the Let's Encrypt team to reduce the impact of this inconvenience at the best of our possibilities.

  3. identified Oct 13, 2021, 10:26 PM UTC

    We inform you that this issue does not affected Generic short URLs redirect (links with rebrand.ly or rb.gy domain name).

  4. monitoring Oct 14, 2021, 01:27 AM UTC

    We are monitoring progress as domain names certificates are generated for the affected domains.

  5. monitoring Oct 14, 2021, 10:36 AM UTC

    HTTPS redirection for affected domain names which were registered via Rebrandly is now fully operational. Monitoring the issuance of SSL certificates for affected domain names which were connected to Rebrandly via DNS.

  6. monitoring Oct 14, 2021, 11:48 AM UTC

    We are continuing to monitor for any further issues.

  7. monitoring Oct 14, 2021, 07:48 PM UTC

    We are aware that, following our resolution announce, there were spare cases where the domain name certificate was issued with delay with respect to the original expectations, or was not issued at all. After a deeper investigation, we observed that today's revocation of a set of SSL certificates uncovered a possible new integration issue between Let's Encrypt and CloudFlare systems. Specifically, customers using proxy services to route their links traffic might be experiencing additional issues at getting their https redirection working again without warnings. We have observed that disablling the active proxy, or alternatively degrading the communication protocol from HTTPS to HTTP for the channel between the Proxy and the Host (Rebrandly) helped exiting this scenario. We will investigate this specific regression outside of the context of this incident.

  8. resolved Oct 14, 2021, 07:48 PM UTC

    This incident has been resolved.