Gainsight experienced a major incident on January 3, 2024 affecting Gainsight CS EU Application, lasting 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 03, 2024, 02:34 PM UTC
We are investigating a sudden increase in error rates which may lead to degraded performance or service interruption.
- monitoring Jan 03, 2024, 02:50 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 03, 2024, 03:14 PM UTC
This incident has been resolved.
- postmortem Feb 23, 2024, 05:43 AM UTC
**Incident:** An isolated number of customers experienced degraded performance in CS-EU Rules on the 3rd of January, 2024. This could have also intermittently impacted the ability to log into the application. **Root Cause:** This incident was result of an elevated number of API requests coming from a single microservice. The unexpected increase led to a build-up of connections, impacting performance on a subset of API servers. Rate limiting functionality was not configured as expected in this case. **Recovery Action:** Once the affected systems and related traffic were identified, Isolating and restarting effected API services resolved the issue immediately. **Preventive Measures:** We have corrected the rate limiter functionality for the microservice that caused this issue.