Voucherify incident
AS1 - Get Redemptions API - degraded performance
Voucherify experienced a minor incident on March 6, 2021 affecting EU1 - API, lasting 2d 15h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 06, 2021, 01:35 AM UTC
We are currently investigating the issue. We have noticed too long response time for some API methods for listing redemptions. This problem occurs only in the AS1 environment which is hosted in the Singapore AWS region.
- identified Mar 06, 2021, 02:28 AM UTC
The issue has been identified and the team applied the temporary fix to solve the problem. Nevertheless, the long-term solution will require us to enhance a few additional things and the final fix will take a bit longer. We will continue monitoring the platform and keep working on a reliable solution for this case.
- identified Mar 06, 2021, 04:42 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Mar 06, 2021, 05:34 AM UTC
A fix has been implemented, and we are monitoring the results. Moreover, we think about the more scalable approach for listing records from databases dedicated to few critical resources - like Redemptions. The team will continue to work on a long-term solution.
- resolved Mar 08, 2021, 04:52 PM UTC
This incident has been resolved. We improved the stability by extending the resources on few databases. The team fully handled the problem. Nevertheless, as part of the fixing and monitoring stage, we identified two functionality elements that require a rethinking of the implementation. The action is required for properly handling the larger scale, which is planned for the second half of this year. Those two functionalities are API methods for publishing unique codes and listing redemptions endpoint. The SRE team analyzed that topic and planned improvements for the near future. Our optimistic estimation assumes that we will release further improvements this week.