Kisi incident

Unusual background job latency

Critical Resolved View vendor source →

Kisi experienced a critical incident on June 9, 2023 affecting Background jobs, lasting 1h 11m. The incident has been resolved; the full update timeline is below.

Started
Jun 09, 2023, 11:41 PM UTC
Resolved
Jun 10, 2023, 12:53 AM UTC
Duration
1h 11m
Detected by Pingoru
Jun 09, 2023, 11:41 PM UTC

Affected components

Background jobs

Update timeline

  1. identified Jun 09, 2023, 11:41 PM UTC

    We have identified an issue with background jobs leading to severely degraded throughput. Outgoing emails, alerts, and webhooks are among the impacted features. The root cause seems to be an outage at one of our service providers (https://status.elastic.co/incidents/07bw653d2677). Our team is working on mitigating the issue.

  2. identified Jun 09, 2023, 11:41 PM UTC

    We are continuing to work on a fix for this issue.

  3. identified Jun 10, 2023, 12:01 AM UTC

    We are working on a fix to mitigate the impact of the service outage. The fix will be deployed shortly.

  4. resolved Jun 10, 2023, 12:53 AM UTC

    Our systems are processing jobs at their normal rate again.