Reown experienced a critical incident on June 30, 2023 affecting Blockchain API, lasting 3h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 30, 2023, 06:40 AM UTC
Our RPC is currently down. Relay and other systems are fully functioning.
- resolved Jun 30, 2023, 09:42 AM UTC
This incident has been resolved.
- postmortem Jun 30, 2023, 09:42 AM UTC
# Summary From ~830am to 1130am CET our RPC was fully down. Reason was that the network as accidentally cut when a VPC was removed. All customers of WalletConnect’s RPC were affected. Customers \(dapps/wallets\) who use another RPC were not affected. # What Happened WalletConnect had sunset their version 1 including the hosted infrastructure. Services were deployed in AWS with only little use of off-the-shelf components including NAT. v1 services were also deployed in a VPC monolith and some new services including the RPC were also deployed in that VPC monolith. When operator removed v1 EC2 instances they unknowingly also removed the NAT preventing the RPC from talking to the outside world. Resolution was to move the RPC service into its own VPC. # What went well * alarms fired immediately * operator was immediately acting # Action items * all services / tooling to move off VPC monolith