Voucherify incident

Potential Platform Issues - sporadic internal server errors

Major Resolved View vendor source →

Voucherify experienced a major incident on May 21, 2021 affecting EU1 - API, lasting 2h 10m. The incident has been resolved; the full update timeline is below.

Started
May 21, 2021, 04:27 PM UTC
Resolved
May 21, 2021, 06:37 PM UTC
Duration
2h 10m
Detected by Pingoru
May 21, 2021, 04:27 PM UTC

Affected components

EU1 - API

Update timeline

  1. investigating May 21, 2021, 04:27 PM UTC

    We are seeing issues on one of our nodes serving responses to API requests. Our Engineering team has started working on mitigation efforts.

  2. investigating May 21, 2021, 04:42 PM UTC

    We are continuing to investigate this issue. However, in the last 50 minutes, we haven't noticed any error which impacts API availability.

  3. resolved May 21, 2021, 06:37 PM UTC

    This incident has been solved. We identified that one of the validation rules due to a bug in a code resulted in an endless loop that was killing individual application nodes. We fixed the problem, and it shouldn't appear anymore. As a next step, we also want to identify how a user added the broken rule to the system and avoid similar cases in the future.