Resolved on June 2, 2025.
The Event Service—Kisi’s core system for capturing and forwarding events—experienced a degradation after a recent feature deployment. An unclosed-connection bug gradually exhausted CPU and memory resources, throttling database writes. Our monitoring detected elevated error rates. A redeployment addressed the bug, but due to a large backlog some delayed event webhooks were unfortunately dropped. However, no events were lost. Service Impact: - Event visibility in the Kisi Admin Dashboard and via API was degraded during the degradation window (new events were not yet persisted). - Event Webhook deliveries and report generation were paused until full recovery. Next Steps: - We will adapt our system to process all event webhooks and event related jobs regardless of backlog size - We will isolate our event pipeline to avoid spillover effects from new functionality
Update timeline
- resolved Jun 09, 2025, 11:27 AM UTC
The Event Service—Kisi’s core system for capturing and forwarding events—experienced a degradation after a recent feature deployment. An unclosed-connection bug gradually exhausted CPU and memory resources, throttling database writes. Our monitoring detected elevated error rates. A redeployment addressed the bug, but due to a large backlog some delayed event webhooks were unfortunately dropped. However, no events were lost. Service Impact: - Event visibility in the Kisi Admin Dashboard and via API was degraded during the degradation window (new events were not yet persisted). - Event Webhook deliveries and report generation were paused until full recovery. Next Steps: - We will adapt our system to process all event webhooks and event related jobs regardless of backlog size - We will isolate our event pipeline to avoid spillover effects from new functionality
Your product runs on more than Kisi
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services