Platform Service Unavailable - Employee Experience Under Maintenance

Update timeline

1. investigating Jan 21, 2025, 08:58 PM UTC

   We are currently investigating reports where the Employee Experience is currently inaccessible due to some sort of maintenance.

2. monitoring Jan 21, 2025, 09:04 PM UTC

   This incident is now resolved. Placing the Employee Experience under monitoring for now.

3. resolved Jan 24, 2025, 03:27 PM UTC

   The impacted components have remained stable and fully available. This incident is now resolved.

4. postmortem Jan 24, 2025, 03:27 PM UTC

   **Summary:** On Tuesday, January 21st, 2025, starting at around 12:35 PM PT, we started receiving reports that the Employee Experience was not visible, but rather, a “Under Construction” notice displayed in its place. Users were unable to close the notice and proceed to the Employee Experience. Initial investigations indicated that the reported issue was further reaching than just the customers who had reported it, and a platform incident was declared and published on the Firstup status page at 12:58 PM PT. **Severity:** Sev2 ‌ **Scope:** The scope of this service disruption included all end users on the Firstup platform who attempted to access the Employee Experience in the duration of this incident. ‌ **Impact:** In the duration of this incident \(30mins\), end users could successfully log into the Employee Experience but saw a “Under Construction” notice instead of their expected Employee Experience frontpage and could not navigate away from the notice overlay. The Employee Experience was still available behind the Pendo guide overlay, and no data was compromised as a result of the incident. ‌ **Root Cause:** The incident response team quickly identified the root cause of the incident to be a Pendo guide overlay that was erroneously published by a Firstup employee across the entire Employee Experience platform at 12:29 PM PT. This guide was intended for a specific customer account, which was not correctly tagged during its publication, and therefore, visible to our entire customer base. Pendo is a third party tool that allows for in-app annotations and user messaging that is primarily used by the Product Management and Marketing organizations primarily targeting program managers actively using Creator Studio. **Mitigation:** To resolve this incident, the Pendo guide was deactivated at 12:59 PM PT, which restored visibility to the Employee Experience immediately. **Recurrence Prevention:** The following actions have been identified as follow-up items to prevent a recurrence of this incident: * Implement a Pendo guide review process before publication. Guides will no longer be immediately able to launch without a review from an authorized publisher. * An audit of all existing guides currently Published should also be carried out, to check for necessary usage and alignment to current standards. * An audit of associate permissions in Pendo to identify who should and should not be able to publish Pendo guides.