[After the Fact] Certificate Updates without SNIs lost SNI mappings for a period of time on November 6th

Update timeline

1. resolved Dec 19, 2025, 03:12 AM UTC

   This is an after-the-fact incident being reported as it was determined that a breaking API change was in production for some time on November 6th, 2025. On November 6 2025, for a period from 15:40 UTC to 19:20 UTC a breaking change to the Konnect API's certificate management endpoints was deployed, causing SNI’s to be disassociated with Certificates if Certificate updates did not include the full SNI list in each update. For users of the Kong Operator, which manages SNIs and Certificates separately and reconciles frequently, this resulted in SNI’s being deleted almost as soon as the API change was deployed to Konnect. Kong has reached out directly to impacted customers.

2. postmortem Dec 19, 2025, 03:12 AM UTC

   # Summary On _**November 6 2025**_, a breaking change to the Konnect API's certificate management endpoints caused SNI’s to be disassociated with Certificates if Certificate updates did not include the full SNI list in each update. ‌ For users of the Kong Operator, which manages SNIs and Certificates separately and reconciles frequently, this resulted in SNI’s being deleted almost as soon as the API change was deployed to Konnect. ## Additional Details The incident originated from an effort to align the Konnect API's behavior with the Kong Gateway API. While Kong Gateway allows Server Name Indicators \(SNIs\) to be managed directly through the Certificate endpoint, Konnect API historically managed certificates and SNIs as separate entities. To achieve consistency, the Konnect team updated the Create and Upsert endpoints to match Kong's behavior. ‌ However, this update introduced an unintended breaking change: when no SNIs were provided in an upsert request, the new logic purged all existing SNIs associated with the certificate instead of leaving them unchanged as in the previous implementation. ‌ This breaking change critically affected the Kong Operator, which manages certificates and SNIs through separate API calls using PUT requests. When the Kong Operator updated certificates without including SNI data, the new Koko logic deleted all associated SNIs. As a result, customer dataplanes fall back to using generic certificates, breaking customer traffic. ‌ ## Kong’s Response The issue was discovered when the CRE team detected service degradation for a customer and alerted the Konnect engineering team. The incident was resolved by reverting the API to its previous version across all regions. ‌ The Kong API does allow for PUT requests to not include every field for a resource. However, in some cases the API must respond with an HTTP400/Bad Request when the lack of some field could result in a destructive action. ‌ Therefore, the Konnect engineering team is adding a mechanism to our API design and testing automation that will block any API that allows critical resource information to be excluded from a request. In this case, the API change would have been blocked, as SNIs would be marked critical to a Certificate request.