k-ID experienced a critical incident on January 27, 2025 affecting Product API and Family Portal and 1 more component, lasting 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 27, 2025, 08:16 PM UTC
The API is degraded due to an expired certificate. Team is replacing the certificate.
- monitoring Jan 27, 2025, 08:28 PM UTC
The expired certificate has been replaced and is being deployed. We expect resolution within the next ten minutes. Also, we identified that the same certificate issue impacts Family Portal and Publisher Portal; the severity of impact has been updated to reflect this.
- monitoring Jan 27, 2025, 08:35 PM UTC
Certificate deployment has completed for all components.
- resolved Jan 27, 2025, 08:38 PM UTC
Traffic flow has returned to normal levels following the replacement of the certificate.
- postmortem Jan 28, 2025, 02:48 AM UTC
Production SSL certificate expired due to an IAM permissions misconfiguration that prevented cert-manager from creating DNS records to verify our domain ownership. * Root cause was a syntax error in the IAM policy that was previously unknown to our product teams. The rule was updated manually with the correct syntax during the incident and has since been rectified in our infrastructure configuration. * We have added an alert for certificates two weeks prior to expiration. This will allow engineers to verify that automated certificate rotations are successful and intervene manually if required before the certificate expires. | **Impact** | **High** | | --- | --- | | Impact duration | 27m | | Time to detect | 2m | | Time to resolve | 25m |