Genea incident

Database - Degraded Performance

Minor Resolved View vendor source →

Genea experienced a minor incident on September 25, 2023 affecting Dashboard and API, lasting 5h 5m. The incident has been resolved; the full update timeline is below.

Started
Sep 25, 2023, 12:45 PM UTC
Resolved
Sep 25, 2023, 05:50 PM UTC
Duration
5h 5m
Detected by Pingoru
Sep 25, 2023, 12:45 PM UTC

Affected components

DashboardAPI

Update timeline

  1. investigating Dec 18, 2023, 11:32 AM UTC

    We've been alerted for a DB spike, and there are delays in APIs, with the dashboard also experiencing slowness.

  2. identified Dec 18, 2023, 11:37 AM UTC

    Multiple customers performed mass actions with card API calls at high frequency (up to 1000 req/min) and mass user sync via Okta, which created a spike on our DB.

  3. monitoring Dec 18, 2023, 11:42 AM UTC

    We have applied rate limits in the list card API and implemented a centralized rate limit for all public APIs. we're continuing to monitor

  4. resolved Dec 18, 2023, 11:44 AM UTC

    The incident has been resolved. Please reach out to Genea Support if you're still experiencing issues.