Gainsight PX incident

PX Engagements and Knowledge Center (KC) Bots

Minor Resolved View vendor source →

Gainsight PX experienced a minor incident on April 25, 2023, lasting 7h 9m. The incident has been resolved; the full update timeline is below.

Started
Apr 25, 2023, 03:28 PM UTC
Resolved
Apr 25, 2023, 10:38 PM UTC
Duration
7h 9m
Detected by Pingoru
Apr 25, 2023, 03:28 PM UTC

Update timeline

  1. identified Apr 25, 2023, 03:28 PM UTC

    Some customers may have issues with PX Engagements or Knowledge Center (KC) Bots. We have identified the cause and we're rolling out a fix. Start time of this issue: 1:00 PM UTC

  2. identified Apr 25, 2023, 07:02 PM UTC

    Our Engineering team is rolling out a fix and the ETA for full resolution is 3 hours.

  3. resolved Apr 25, 2023, 10:38 PM UTC

    This incident has been resolved.

  4. postmortem May 02, 2023, 10:51 PM UTC

    **Summary of Incident:** Customers using PX, either as a customer or consumer, were unable to use Engagements and KC Bots starting on April 25, 2023 6:30 am PDT that was resolved by 15:38 PDT. ‌ **Root Cause:** The April 25 incident was caused by two issues: 1. A caching cluster slow-down. This cluster is what serves the KC Bot and Engagements portion of the product 2. Intermittent KC Bot synchronization issues due to malformed data ‌ **Immediate Action Items:** 1. Caching Cluster sizing and configuration adjustments 2. KC Bot Configuration adjustments and a protection mechanism for malformed data issues ‌ Please reach out to [[email protected]](mailto:[email protected]) if you have questions.