Kong Outage History

This incident has been resolved.

Customers running 3.13 and below who use the OIDC plugin with `tls_client_auth_ssl_verify` unset would have seen this value change to `false` if they updated the config after the rollback of new defaults following the 3.14 release. The rollback incorrectly flipped the oidc plugin tls_client_auth_ssl_verify to false as a default, which was not one of the items recently switched to default true and has instead been defaulted true for some time. We have rolled out a fix to prod to change this default back to true. Updates to the OIDC plugin should once again keep this value set to true if not specifically defined.

We have completed the rollback to the original default values. Customers applying their configurations without explicitly defining ssl_verify and hide_credentials will default to `false` again.

On 4/2/2026. Konnect introduced a change as part of our 'secure by default' initiatives for the upcoming 3.14 release, which caused expected_hash syncing issues for some customers' data planes. For background, when a data plane connects to the Konnect control plane, the configuration is produced for that specific version of the Kong gateway. A configuration hash is calculated based off of the produced config, and stored as the expected config hash. When configuration changes occur, the control plane calculates the new configuration, broadcasts it to the connected data planes, and updates to the new expected hash. The issue here was the preparation of the control plane for supporting SHA256 to store credential hashes. When this change was introduced, it resulted in a minor structural difference in the produced configuration, but no actual change to the configurations occurred. This led to the same configuration yielding a different hash before and after the deploy. The way this could get a data plane to report 'out of sync' was the following: - A data plane is connected and in sync with DP hash and Expected Config Hash `a111111a` - The data plane is disconnected/reconnected, and since it is still in sync with DP hash and Expected Config Hash `a111111a`, a new config is not calculated - The change to embed SHA256 support in the control plane is deployed - A new data plane connects with no config loaded, and reports an empty config to the Control Plane - The control plane generates the slightly altered config, and ends up with a new config hash, which updates the expected config hash to `b22222b` - Since no configuration change occurred, the data planes already connected and on `1aaaaaa1` would not have been pushed a change event, and would still be running the same configuration as `b222222b` but the hash would not match, causing the DPs to appear out of sync Note: Once *any* change to the control plane configuration occurred, a new config event would have been delivered, and all previous data planes would have been updated to the hash based on the new structure. Additionally, if an impacted data plane were to disconnect and reconnect, they also would have been brought into sync with the new hash. The issue was the system failing to consider the new hash as an 'event' since it did not change configured proxy behavior. To address this in the short term, the Konnect engineering team has ensured structural changes like this which result in a config hash will trigger a configuration change event, even when no config has changed. To address this in the long term, Konnect is moving to a config version tracking methodology that will not be impacted by structural, non-behavioral changes as the hash is.

This incident has been resolved.

On 4/2/2026. Konnect introduced a change as part of our 'secure by default' initiatives for the upcoming 3.14 release, which caused expected_hash syncing issues for some customers' data planes. For background, when a data plane connects to the Konnect control plane, the configuration is produced for that specific version of the Kong gateway. A configuration hash is calculated based off of the produced config, and stored as the expected config hash. When configuration changes occur, the control plane calculates the new configuration, broadcasts it to the connected data planes, and updates to the new expected hash. The issue here was the preparation of the control plane for supporting SHA256 to store credential hashes. When this change was introduced, it resulted in a minor structural difference in the produced configuration, but no actual change to the configurations occurred. This led to the same configuration yielding a different hash before and after the deploy. The way this could get a data plane to report 'out of sync' was the following: - A data plane is connected and in sync with DP hash and Expected Config Hash `a111111a` - The data plane is disconnected/reconnected, and since it is still in sync with DP hash and Expected Config Hash `a111111a`, a new config is not calculated - The change to embed SHA256 support in the control plane is deployed - A new data plane connects with no config loaded, and reports an empty config to the Control Plane - The control plane generates the slightly altered config, and ends up with a new config hash, which updates the expected config hash to `b22222b` - Since no configuration change occurred, the data planes already connected and on `1aaaaaa1` would not have been pushed a change event, and would still be running the same configuration as `b222222b` but the hash would not match, causing the DPs to appear out of sync Note: Once *any* change to the control plane configuration occurred, a new config event would have been delivered, and all previous data planes would have been updated to the hash based on the new structure. Additionally, if an impacted data plane were to disconnect and reconnect, they also would have been brought into sync with the new hash. The issue was the system failing to consider the new hash as an 'event' since it did not change configured proxy behavior. To address this in the short term, the Konnect engineering team has ensured structural changes like this which result in a config hash will trigger a configuration change event, even when no config has changed. To address this in the long term, Konnect is moving to a config version tracking methodology that will not be impacted by structural, non-behavioral changes as the hash is.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

From 15:25z to 15:35z, the Konnect Control Plane saw an increase in dropped connections and re-connections in the US region from connected dataplanes. Dataplanes automatically re-connect when a connection is dropped, so there should be no observed impact, but customers may have noticed an increase in log volume if re-connections were occurring during this window.

The issue has been resolved. Delivery is now operating as expected.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.