Ketch had 9 outages in the last 2 years totaling 145h 27m of downtime — averaging 0.4 incidents per month.
There were 9 Ketch outages since September 18, 2024 totaling 145h 27m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are investigating elevated latency and intermittent errors affecting consent API requests in our EU (Frankfurt) region. We are increasing database capacity and will provide an update shortly.
The orchestration of marketing preferences experienced an outage between 9:47 am PST and 10:05 am PST on January 6, 2016. During this period, orchestrations for downstream systems concerning market preferences were not executed.
Summary We are experiencing processing delays in Marketing Preferences workflows due to unusually high orchestration load. This is causing some subscription updates to be delayed or time out instead of processing within the usual window. Impact - A subset of Marketing Preferences workflows (including some customer webhooks) did not complete within expected timeframes. - No data has been lost; all incoming events are safely stored, but some have not yet been orchestrated to downstream destinations. No action is required. If you rely on webhooks or downstream reporting from Marketing Preferences, you may see a burst of historical events as we work through the backlog.
We are continuing to monitor for any further issues.
This incident has been resolved.
We're experiencing an elevated level of errors since 10:11 AM Pacific and are currently looking into the issue.
We are continuing to investigate this issue.
The issue has been identified, a fix implemented, and services are now recovering.
This incident has been resolved.
We experienced elevated API error rates due to a distributed denial-of-service (DDoS) attack. Mitigations have been put in place, and the team will monitor and adjust as necessary.
This incident has been resolved.
At 9:30 AM Pacific Time, we observed elevated errors for some components within the Ketch Application Console. The engineering teams were promptly engaged, and the service was successfully restored by 10:08 AM Pacific Time.
At 6:10 AM Pacific Time, an increased incidence of API errors was observed concerning DSR requests. The engineering teams were promptly engaged, and service was successfully restored at 10:08 AM Pacific Time.
The issue has been identified and engineers are working on restoring service.
We are continuing to restore services in the EU region.
Services are starting to recover, and the recovery is being actively monitored.
This incident has been resolved. Ketch Engineering will continue to monitor the situation.
Ketch Management Console not Loading.
Between 7:42 AM and 10:30 AM \(Pacific Time\) on September 18th, 2024, some users experienced slow or unresponsive performance from the Ketch Management Console. Engineers were notified about the service degradation and discovered that an ad-hoc backend task was unexpectedly causing high load, which led to the degraded performance of the Ketch Console. Ketch Engineering took respective remediation steps to restore service to the Ketch Management Console. Services were continually monitored and confirmed back to normal latency levels by 10:30 AM. This incident was isolated to the Ketch Management Console and did not impact any other APIs. Ketch uses automated monitoring of its systems to detect such incidents; however, steps are being made to further enhance monitoring to detect performance degradations sooner. The team is also adding additional controls to its backend tasks to ensure that this failure scenario does not reoccur.