Ketch incident

Elevated API Errors

Minor Resolved View vendor source →

Ketch experienced a minor incident on May 22, 2022 affecting API (US), lasting 12h 58m. The incident has been resolved; the full update timeline is below.

Started
May 22, 2022, 03:52 AM UTC
Resolved
May 22, 2022, 04:51 PM UTC
Duration
12h 58m
Detected by Pingoru
May 22, 2022, 03:52 AM UTC

Affected components

API (US)

Update timeline

  1. investigating May 22, 2022, 03:52 AM UTC

    We're experiencing an elevated level of API errors and are currently looking into the issue.

  2. investigating May 22, 2022, 04:05 AM UTC

    We are continuing to investigate this issue.

  3. investigating May 22, 2022, 04:34 AM UTC

    We are continuing to investigate this issue

  4. investigating May 22, 2022, 05:35 AM UTC

    The issue has been identified and remediation is in progress

  5. investigating May 22, 2022, 07:06 AM UTC

    We are continuing to investigate this issue.

  6. investigating May 22, 2022, 07:09 AM UTC

    The engineering team has found the cause of this issue and is undertaking remediation efforts to restore capacity.

  7. monitoring May 22, 2022, 08:04 AM UTC

    A fix has been implemented and we are monitoring the results.

  8. monitoring May 22, 2022, 08:24 AM UTC

    We are actively monitoring the recovery of services. Data continues to be ingested, however, some permit data may be delayed while resynchronization efforts proceed.

  9. resolved May 22, 2022, 04:51 PM UTC

    This incident has been resolved.