Sticky incident
High load on one database cluster ("pair 5") affecting API latency and performance
Sticky experienced a minor incident on June 16, 2020 affecting Transaction API and Membership API, lasting 2h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 16, 2020, 04:56 PM UTC
We are looking into some high volume API activity coming into one of our backend database clusters which has been affecting client instances running on one specific database. API response times have been increasing since about 5am ET today, June 16, 2020. We have put in place some hard database concurrent connection limits on some clients in order to temporarily mitigate the performance issues. If you are experiencing abnormal API latency today or hard API response errors, our support team can verify that your instance is on the "pair 5" cluster. We are actively working on monitoring and determining the root cause of the issue.
- monitoring Jun 16, 2020, 06:21 PM UTC
We have identified some offending API users causing the spike in activity and have placed some concurrency restrictions on those specific accounts. This has stabilized the activity on the "pair 5" database cluster and performance metrics for the last 60 minutes have been within normal thresholds. We will continue to monitor this incident.
- resolved Jun 16, 2020, 07:46 PM UTC
This incident has been resolved.