BlockChyp experienced a minor incident on July 20, 2023 affecting Core API, lasting 1h 59m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 20, 2023, 07:39 PM UTC
A sudden increase in demand is causing a temporary elevation of error rates. We're increasing backend throughput capacity with our upstream provider by 8X and error rates should return to normal shortly.
- monitoring Jul 20, 2023, 07:52 PM UTC
Capacity has been increased and error rates appear to be coming down.
- monitoring Jul 20, 2023, 09:31 PM UTC
We've doubled memory in the api cluster to aid in recovery. We're seeing an issue with duplicate store and forward transactions and these retries are causing a lot of traffic. We're investigating the store and forward issue, but response times seem to be coming back down. We've seen no transaction errors for the last few minutes.
- resolved Jul 20, 2023, 09:39 PM UTC
Error rates and performance rates have returned to baseline.