Kisi incident

Post mortem: June 16 elevated error rates

Notice Resolved View vendor source →

Kisi experienced a notice incident on June 26, 2025 affecting Endpoints, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jun 26, 2025, 10:55 AM UTC
Resolved
Jun 26, 2025, 10:55 AM UTC
Duration
Detected by Pingoru
Jun 26, 2025, 10:55 AM UTC

Affected components

Endpoints

Update timeline

  1. resolved Jun 26, 2025, 10:55 AM UTC

    On June 16, 2025, the Kisi production API experienced a temporary degradation in availability. Investigation revealed that the root cause was an unexpected behavior in Google Cloud Platform’s (GCP) Cloud Run during an internal stack migration and service redistribution process managed by GCP. According to GCP’s engineering response, a background migration and redistribution of workloads triggered a one-time instance restart in our Cloud Run service. This resulted in the application falling below its configured minimum instances temporarily, impacting our ability to serve requests reliably. While the redistribution process is part of GCP’s infrastructure optimization efforts, the side effect of under-scaling was not communicated in advance and not within our control. GCP has confirmed that a fix has been rolled out to address this issue and prevent recurrence in similar scenarios.