RRPproxy experienced a major incident on April 29, 2021 affecting XRRP API and EPP API and 1 more component, lasting 1h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 29, 2021, 08:44 PM UTC
We are currently investigating this issue.
- identified Apr 29, 2021, 09:19 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Apr 29, 2021, 10:21 PM UTC
This incident has been resolved.
- postmortem Apr 30, 2021, 10:22 AM UTC
# Statement for API outage on April 29, 2021 ## Description We had Database issues on April 29, 2021 between 17:12 UTC and 23:14 UTC. As a result our RRPproxy API responded with errors and was partially unreachable during this timeframe. ## Details ### 17:12 UTC to 19:39 UTC Responses to writing Commands \(such as AddDomain, ModifyDomain, ...\) sent to our API began to show an error: code = 549 description = Command failed; Internal server error \(create job\) ### 19:40 UTC to 22:21 UTC Too many Commands piled up and as a result our API wasn't reachable any more. Investigation showed that a database index for a table has become too big. Subsequently the index has been increased accordingly. Afterwards the API reachability was restored. ### 22:22 UTC to 23:14 UTC Partially issues with our XRRP and EPP API gateways which resulted as an aftermath. ## Impact on Security None. No data has been corrupted. All domains and services continued to resolve. ## Follow Up Actions Monitoring of maximum Index Values for primary Keys on all Database Tables has been put in place.