Sardine incident

Elevated latency for EU instance API

Resolved after 50 minutes, affecting Issuing API and Customer APIs, on June 2, 2026.

postmortem Jun 08, 2026, 08:41 PM UTC

### Summary On June 2, 2026, some customers experienced elevated latency on `/v1/issuing/risks` and `/v1/customers` requests in `prod-eu`. We’re sorry for the disruption and for any delays this caused in customer risk decisions. ### What Happened A subset of issuing risk requests became slower than expected due to delayed customer data lookups. During the main impact window, affected requests could take up to approximately 5 seconds before continuing through a soft-fail path. ### Why It Happened The issue was caused by a customer lookup path not enforcing its intended short timeout. When some customer lookups stalled, they waited on the broader request deadline instead of failing fast at the operation-level timeout. ### What We’re Doing About It We deployed a hotfix to reduce the customer-load timeout and limit how long issuing requests can wait on stalled lookups. We are also adding follow-up improvements to reduce Datastore dependency in low-latency paths and investigate the underlying cause of the lookup stalls.

Started
Jun 02, 2026, 05:43 PM UTC
Resolved
Jun 02, 2026, 06:34 PM UTC
Duration
50m
Detected by Pingoru
Jun 02, 2026, 05:43 PM UTC

Affected components

Issuing APICustomer APIs

Update timeline

  1. investigating Jun 02, 2026, 05:43 PM UTC

    We are currently investigating this issue.

  2. resolved Jun 02, 2026, 06:34 PM UTC

    This incident has been resolved.

  3. postmortem Jun 08, 2026, 08:41 PM UTC

    ### Summary On June 2, 2026, some customers experienced elevated latency on `/v1/issuing/risks` and `/v1/customers` requests in `prod-eu`. We’re sorry for the disruption and for any delays this caused in customer risk decisions. ### What Happened A subset of issuing risk requests became slower than expected due to delayed customer data lookups. During the main impact window, affected requests could take up to approximately 5 seconds before continuing through a soft-fail path. ### Why It Happened The issue was caused by a customer lookup path not enforcing its intended short timeout. When some customer lookups stalled, they waited on the broader request deadline instead of failing fast at the operation-level timeout. ### What We’re Doing About It We deployed a hotfix to reduce the customer-load timeout and limit how long issuing requests can wait on stalled lookups. We are also adding follow-up improvements to reduce Datastore dependency in low-latency paths and investigate the underlying cause of the lookup stalls.

Your product runs on more than Sardine

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →