BlockChyp experienced a major incident on July 19, 2023 affecting Core API, lasting 2h 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 19, 2023, 08:12 PM UTC
BlockChyp is currently experiencing elevated error rates due to a memory issue in the cloud relay routing system. A fix is being tested and will be deployed shortly.
- monitoring Jul 19, 2023, 08:50 PM UTC
A fix has been deployed and we are monitoring the recovery.
- monitoring Jul 19, 2023, 09:34 PM UTC
The memory issue appears to be fixed. Response times and timeouts have improved, but we're still seeing higher than normal error rates. We believe this is related to fixed concurrent traffic limits imposed by our backend processing partners and are working to increase capacity.
- monitoring Jul 19, 2023, 10:09 PM UTC
We've increased our concurrent transaction capacity with back end partners. We're seeing response times start to come down. The avg is the lowest it's been since the incident began.
- monitoring Jul 19, 2023, 10:16 PM UTC
Current response time avg is down to 1.38 seconds, which is in the normal band. We'll continue monitoring a bit longer and ensure the system remains stable.
- resolved Jul 19, 2023, 10:30 PM UTC
The avg response time for the last 15 minutes is now 1.22 seconds. According to our logs, the incident effectively stopped immediately after we applied the back end capacity changes, which was at 4:09 PM, Mountain Time. Thanks to everyone for their patience. We have no changes planned for tomorrow.