Increased error rates loading web experience across regions...

Update timeline

1. investigating Oct 06, 2026, 08:20 PM UTC

   We are currently investigating an elevated error rate when loading employee experience causing slow page loads and intermittent timeouts.

2. identified Oct 06, 2026, 08:24 PM UTC

   Root cause has been identified. Impact on US1 region should be mitigated, and full recovery in process. US2 and EU regions are being investigated now.

3. identified Oct 06, 2026, 08:32 PM UTC

   EU region impact mitigated and fully recovered. Continuing to monitor region health. Work continues to mitigate impact on US2.

4. identified Oct 06, 2026, 09:00 PM UTC

   New user sessions cannot be created still on US2 region, but other services have recovered. Users with existing active sessions should be able to access and interact with programs hosted on that region.

5. monitoring Oct 06, 2026, 09:15 PM UTC

   Impact on US2 has been mitigated, and remaining services are fully operational. Actively monitoring all regions to ensure steady state.

6. monitoring Oct 06, 2026, 09:37 PM UTC

   Monitoring has confirmed all regions remain health, and the incident has now been fully resolved.

7. resolved Oct 06, 2026, 10:47 PM UTC

   This incident has been resolved.

8. postmortem Oct 14, 2026, 09:13 PM UTC

   **Summary:** On October 6th, 2026, at 1:00 PM PT, our monitoring alerted us to a high number of connection errors in the production environments shortly after a hotfix was deployed. Preliminary engineering investigations coupled with internal and external reports of platform loading errors confirmed platform reachability issues. A platform incident was declared at 1:11 PM PT, and the hotfix was rolled back, along with a targeted code fix deployment to address the underlying code error. All services were fully restored by 2:09 PM PT the same day. **Severity:** Sev-1 **Scope:** All Firstup customers were affected, with customers in US2 and EU platforms experiencing a more prolonged service disruption due to circular restart dependencies that had to be addressed separately from the hot fix rollback. Customers in the US1 platform experienced a brief disruption of service primarily accompanied by degraded application responsiveness. **Impact:** * **US1:** From 1:00 PM PT to 1:19 PM PT, service was degraded and the platform processed up to 70% less work than normal. * **US2:** From 1:00 PM PT to 2:09 PM PT \(69 minutes\), users couldn't log in to Studio, Employee Experience or the support portal, and extended message delivery times. * **EU:** From 1:00 PM PT to 1:26 PM PT \(26 minutes\), users had the same problems as in US2. **Root Cause:** The root cause for this incident was identified to be the deployment of a hotfix that was built on an updated shared base software image. That image included newer networking and encryption libraries that don't work with the web server setup of one of our core services, so the service crashed. While this problem had already been identified and protected against in our regular release processes, it had not been protected against in the hotfix deployment process branch. Our systems and services health checks didn't catch the failure until the healthy copies of the impacted service had already been replaced. Additionally, startup dependencies between two backend services increased the time to recovery and was more pronounced in the US2 platform. **Mitigation:** In US1 and EU, the deployed hotfix was rolled back and services were restored by 1:19 PM PT and 1:26 PM PT, respectively. In US2, and in addition to the hotfix rollback, we deployed a code fix that: * removed the startup dependency between the two backend services * locked the base software image to the last working version This fully restored service by 2:09 PM PT. The same fix then went to all regions in a scheduled release at 6:50 PM PT on October 6th, 2026, with no customer impact. **Recurrence Prevention:** To prevent this incident from happening again due to the same root cause, Firstup has done \(or committed to doing\) the following: * Lock and version shared base software images in every build process, including hotfixes, so that what is test is exactly what is released. * Roll out changes in stages, starting with one region or part of capacity, so problems are caught before they reach every region. * Strengthen health checks so they confirm the service is actually working before it takes traffic. * Removed the identified startup dependencies between the two core services in the US1 and EU platforms by deploying the code fix during our maintenance window on October 6th, 2026. * Conduct a review of any remaining dependencies for similar risks. * Tighten the hotfix process. Hotfixes that aren't deployed promptly will need re-approval, and related changes across services will be deployed together. * Update monitoring and recovery runbooks in every production region.