Is Firstup down?

We are currently investigating an elevated error rate when loading employee experience causing slow page loads and intermittent timeouts.

Root cause has been identified. Impact on US1 region should be mitigated, and full recovery in process. US2 and EU regions are being investigated now.

EU region impact mitigated and fully recovered. Continuing to monitor region health. Work continues to mitigate impact on US2.

New user sessions cannot be created still on US2 region, but other services have recovered. Users with existing active sessions should be able to access and interact with programs hosted on that region.

Impact on US2 has been mitigated, and remaining services are fully operational. Actively monitoring all regions to ensure steady state.

Monitoring has confirmed all regions remain health, and the incident has now been fully resolved.

This incident has been resolved.

**Summary:** On October 6th, 2026, at 1:00 PM PT, our monitoring alerted us to a high number of connection errors in the production environments shortly after a hotfix was deployed. Preliminary engineering investigations coupled with internal and external reports of platform loading errors confirmed platform reachability issues. A platform incident was declared at 1:11 PM PT, and the hotfix was rolled back, along with a targeted code fix deployment to address the underlying code error. All services were fully restored by 2:09 PM PT the same day. **Severity:** Sev-1 **Scope:** All Firstup customers were affected, with customers in US2 and EU platforms experiencing a more prolonged service disruption due to circular restart dependencies that had to be addressed separately from the hot fix rollback. Customers in the US1 platform experienced a brief disruption of service primarily accompanied by degraded application responsiveness. **Impact:** * **US1:** From 1:00 PM PT to 1:19 PM PT, service was degraded and the platform processed up to 70% less work than normal. * **US2:** From 1:00 PM PT to 2:09 PM PT \(69 minutes\), users couldn't log in to Studio, Employee Experience or the support portal, and extended message delivery times. * **EU:** From 1:00 PM PT to 1:26 PM PT \(26 minutes\), users had the same problems as in US2. **Root Cause:** The root cause for this incident was identified to be the deployment of a hotfix that was built on an updated shared base software image. That image included newer networking and encryption libraries that don't work with the web server setup of one of our core services, so the service crashed. While this problem had already been identified and protected against in our regular release processes, it had not been protected against in the hotfix deployment process branch. Our systems and services health checks didn't catch the failure until the healthy copies of the impacted service had already been replaced. Additionally, startup dependencies between two backend services increased the time to recovery and was more pronounced in the US2 platform. **Mitigation:** In US1 and EU, the deployed hotfix was rolled back and services were restored by 1:19 PM PT and 1:26 PM PT, respectively. In US2, and in addition to the hotfix rollback, we deployed a code fix that: * removed the startup dependency between the two backend services * locked the base software image to the last working version This fully restored service by 2:09 PM PT. The same fix then went to all regions in a scheduled release at 6:50 PM PT on October 6th, 2026, with no customer impact. **Recurrence Prevention:** To prevent this incident from happening again due to the same root cause, Firstup has done \(or committed to doing\) the following: * Lock and version shared base software images in every build process, including hotfixes, so that what is test is exactly what is released. * Roll out changes in stages, starting with one region or part of capacity, so problems are caught before they reach every region. * Strengthen health checks so they confirm the service is actually working before it takes traffic. * Removed the identified startup dependencies between the two core services in the US1 and EU platforms by deploying the code fix during our maintenance window on October 6th, 2026. * Conduct a review of any remaining dependencies for similar risks. * Tighten the hotfix process. Hotfixes that aren't deployed promptly will need re-approval, and related changes across services will be deployed together. * Update monitoring and recovery runbooks in every production region.

We are currently investigating multiple reports of shortcuts not rendering as expected for programs hosted on US2. No other regions are impacted at this time.

We have identified a potential root cause for this issue and are working to mitigate it. Another update will be provided within 1 hour.

A fix for this issue has been released. A refresh of the Employee Experience page may be required to reload the shortcuts. We will be placing the impacted components under monitoring for now.

This incident has been resolved, and all impacted components have remained stable and fully available. A Root Cause Analysis for this incident will be provided as soon as the incident postmortem has been concluded.

We are currently investigating reports where Insights Reports are intermittently failing to load for EU customers only. We will provide an update within 1 hour.

We have identified a potential root cause for this issue and are working towards mitigation. Another update will be provided within the next 1 hour.

We have fixed the underlying issue with loading Insights Reports and completed a data backfill task to restore any data that may have been missed in the reports. We will be placing the affected components under monitoring for now.

Insights Reports have remained stable and fully available after the fix was released in the production environment. This incident is now resolved. We will publish a Root Cause Analysis here once a full incident postmortem has been completed.

We are currently investigating a delay in data populating in Insights reports. Reports in scope include the Campaign Results Page, Campaign Performance, Videos overview, hover popup modal, Campaign Delivery, and possibly others.

The issue has been identified that has caused a backup of data pipeline processing and are in the process of working through the data backfill. We expect reporting data to be fully caught up within the next 2 hours with incremental improvements already visible now.

We are continuing to actively monitor the data backfill progress and remain on target for full restoration of normal processing times by 1PM PT. Only programs on the US1 region are in scope of the issue.

There have been a few unexpected slowdowns in the backfill processing and we now expect the completion to be at least another hour from now. Also, not originally considered in scope, Poll reporting is also affected and being backfilled along with other reporting data.

All processing for events that occurred on 8/12 has concluded. There are events on 8/13 that are still being processed as that backfill is competing against new events coming in today. As a result, the backfill processing for today's events has had to be restarted several times-- adding unexpected delay to the completion time. Next update will be 1 hour from now, or following the next milestone completion, whichever is sooner.

Backfill process continuing to run for 08/13 events. Blocker has been removed that required multiple restarted and we have more confidence now in the throughput of the backfill process that will be complete for 08/13 within the hour. Focus will then shift to historic events on 08/10 and 08/11 which are also in scope.

Backfill for 08/13 complete and we are now back at normal operating cadence for new data being ingested. Dashboards are all up to date, and campaign delivery reports, and hoverover reports from the campaign results page are in the process of being refreshed.

This incident is now resolved and all affected components marked fully operational.

**Summary:** On August 12th, 2026, a system monitor alerted us of a potential issue within our metrics data processing pipelines. Preliminary engineering investigations coupled with internal and external reports of an issue with Insights Reports revealed a wider scope and impact. A platform incident was declared on August 13th, 2026, at 16:33 UTC, and an incident response team was assembled. After determining the cause of the unprocessed data backlog and mitigating its impact, data backfill resumed on August 13th, 2026, at 23:22 UTC, eventually completing, and service was fully restored on August 14th, 2026, at 02:31 UTC. **Severity:** Sev-2 **Scope:** Between August 12th, 2026, at 10:00 UTC and August 14th, 2026, at 02:31 UTC \(40 hours and 31 minutes\), Studio users in the Firstup US1 environment were unable to reliably utilize Insights Reports. Users in the US2 and EU environments were not in the scope of this incident. **Impact:** Studio users in the Firstup US1 environment observed loading issues and stale data in Studio Insights Reports. Insights Reports in the US2 and EU environments were not impacted by this incident. **Root Cause:** The primary root cause was identified to be a software defect due to missing input data validation from the Employee Experience \(EE\) client side to the data ingestion layer. The client-side _time\_spent\_reading_ property accumulated active user reading duration without a cap or periodic flush, emitting only when the page/article closed. Any user reading session remaining continuously active for ~24.86 days produced a value exceeding the allowed integer boundary value, causing an integer overflow to downstream data pipelines that were not designed to handle integer overflows. As a resulted, the data pipelines failed, causing Insights Reports to stop ingesting new metrics. **Mitigation:** To stabilize and remediate the data pipeline, the following steps were taken: * **Truncation at Ingest:** A truncation filter was applied at the ingestion layer to cap incoming _time\_spent\_reading_ values, halting new integer overflows from reaching downstream pipelines. * **Data Sanitization:** Existing overflow values in _real\_time\_events\_details_ were manually reduced to allow transformation pipelines to process records. * **Lock Resolution:** An underlying data table lock on improperly terminated processes was identified and cleared, allowing frozen backfills to resume. * **Backfill & Schedule De-confliction:** Automated scheduled tasks that were running concurrently with the manual backfill scripts were resolved, allowing campaign metrics and integration data backfills to complete successfully. **Recurrence Prevention:** To prevent recurrence and improve resilience, Firstup has committed to the following actions: * **Input Sanitation & Capping:** Implement input restrictions on _time\_spent\_reading_ events from the Employee Experience \(EE\) client side, and on the data ingestion layer to cap values to 1 hour \(3,600,000 ms\) before they enter the data processing pipelines. * **Playbook Expansion:** Update the Engineering Playbook with explicit guidance on identifying/clearing process locks, disabling scheduled jobs during manual backfills, and running backfill scripts. * **Pipeline Resilience & Graceful Error Handling:** Evaluate per-record error handling and modular pipeline refactoring to prevent an isolated invalid record from halting global pipeline processing across all customer programs.