Kisi incident

Increased error rate

Minor Resolved View vendor source →

Kisi experienced a minor incident on June 27, 2022 affecting Endpoints and Background jobs, lasting 2h 53m. The incident has been resolved; the full update timeline is below.

Started
Jun 27, 2022, 06:04 PM UTC
Resolved
Jun 27, 2022, 08:58 PM UTC
Duration
2h 53m
Detected by Pingoru
Jun 27, 2022, 06:04 PM UTC

Affected components

EndpointsBackground jobs

Update timeline

  1. investigating Jun 27, 2022, 09:49 AM UTC

    We have noticed an increased number of errors in both background jobs and endpoints. We are currently investigating.

  2. investigating Jun 27, 2022, 09:49 AM UTC

    We are continuing to investigate this issue.

  3. investigating Jun 27, 2022, 10:14 AM UTC

    The issue seems to be related to some scheduled database maintenance. We have taken some temporary measures to ensure API availability and we are working on identifying the root cause.

  4. identified Jun 27, 2022, 11:08 AM UTC

    Most API calls are unaffected. The unlock failure rate is currently normal. We have identified the issue and are working on a fix.

  5. identified Jun 27, 2022, 12:59 PM UTC

    We will deploy a code change that addresses this issue within an hour.

  6. monitoring Jun 27, 2022, 05:04 PM UTC

    A fix has been deployed and we are monitoring the results.

  7. resolved Jun 27, 2022, 08:58 PM UTC

    A fix has been deployed and we are monitoring the results.