Ketch experienced a minor incident on September 18, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Sep 18, 2024, 08:35 PM UTC
Ketch Management Console not Loading.
- postmortem Sep 19, 2024, 10:34 PM UTC
Between 7:42 AM and 10:30 AM \(Pacific Time\) on September 18th, 2024, some users experienced slow or unresponsive performance from the Ketch Management Console. Engineers were notified about the service degradation and discovered that an ad-hoc backend task was unexpectedly causing high load, which led to the degraded performance of the Ketch Console. Ketch Engineering took respective remediation steps to restore service to the Ketch Management Console. Services were continually monitored and confirmed back to normal latency levels by 10:30 AM. This incident was isolated to the Ketch Management Console and did not impact any other APIs. Ketch uses automated monitoring of its systems to detect such incidents; however, steps are being made to further enhance monitoring to detect performance degradations sooner. The team is also adding additional controls to its backend tasks to ensure that this failure scenario does not reoccur.