Insights Reporting Data Delays

Update timeline

1. investigating Aug 13, 2026, 05:06 PM UTC

   We are currently investigating a delay in data populating in Insights reports. Reports in scope include the Campaign Results Page, Campaign Performance, Videos overview, hover popup modal, Campaign Delivery, and possibly others.

2. identified Aug 13, 2026, 05:59 PM UTC

   The issue has been identified that has caused a backup of data pipeline processing and are in the process of working through the data backfill. We expect reporting data to be fully caught up within the next 2 hours with incremental improvements already visible now.

3. monitoring Aug 13, 2026, 06:59 PM UTC

   We are continuing to actively monitor the data backfill progress and remain on target for full restoration of normal processing times by 1PM PT. Only programs on the US1 region are in scope of the issue.

4. monitoring Aug 13, 2026, 08:15 PM UTC

   There have been a few unexpected slowdowns in the backfill processing and we now expect the completion to be at least another hour from now. Also, not originally considered in scope, Poll reporting is also affected and being backfilled along with other reporting data.

5. monitoring Aug 13, 2026, 09:39 PM UTC

   All processing for events that occurred on 8/12 has concluded. There are events on 8/13 that are still being processed as that backfill is competing against new events coming in today. As a result, the backfill processing for today's events has had to be restarted several times-- adding unexpected delay to the completion time. Next update will be 1 hour from now, or following the next milestone completion, whichever is sooner.

6. monitoring Aug 13, 2026, 10:41 PM UTC

   Backfill process continuing to run for 08/13 events. Blocker has been removed that required multiple restarted and we have more confidence now in the throughput of the backfill process that will be complete for 08/13 within the hour. Focus will then shift to historic events on 08/10 and 08/11 which are also in scope.

7. monitoring Aug 14, 2026, 12:38 AM UTC

   Backfill for 08/13 complete and we are now back at normal operating cadence for new data being ingested. Dashboards are all up to date, and campaign delivery reports, and hoverover reports from the campaign results page are in the process of being refreshed.

8. resolved Aug 14, 2026, 02:31 AM UTC

   This incident is now resolved and all affected components marked fully operational.

9. postmortem Sep 14, 2026, 05:16 PM UTC

   **Summary:** On August 12th, 2026, a system monitor alerted us of a potential issue within our metrics data processing pipelines. Preliminary engineering investigations coupled with internal and external reports of an issue with Insights Reports revealed a wider scope and impact. A platform incident was declared on August 13th, 2026, at 16:33 UTC, and an incident response team was assembled. After determining the cause of the unprocessed data backlog and mitigating its impact, data backfill resumed on August 13th, 2026, at 23:22 UTC, eventually completing, and service was fully restored on August 14th, 2026, at 02:31 UTC. **Severity:** Sev-2 **Scope:** Between August 12th, 2026, at 10:00 UTC and August 14th, 2026, at 02:31 UTC \(40 hours and 31 minutes\), Studio users in the Firstup US1 environment were unable to reliably utilize Insights Reports. Users in the US2 and EU environments were not in the scope of this incident. **Impact:** Studio users in the Firstup US1 environment observed loading issues and stale data in Studio Insights Reports. Insights Reports in the US2 and EU environments were not impacted by this incident. **Root Cause:** The primary root cause was identified to be a software defect due to missing input data validation from the Employee Experience \(EE\) client side to the data ingestion layer. The client-side _time\_spent\_reading_ property accumulated active user reading duration without a cap or periodic flush, emitting only when the page/article closed. Any user reading session remaining continuously active for ~24.86 days produced a value exceeding the allowed integer boundary value, causing an integer overflow to downstream data pipelines that were not designed to handle integer overflows. As a resulted, the data pipelines failed, causing Insights Reports to stop ingesting new metrics. **Mitigation:** To stabilize and remediate the data pipeline, the following steps were taken: * **Truncation at Ingest:** A truncation filter was applied at the ingestion layer to cap incoming _time\_spent\_reading_ values, halting new integer overflows from reaching downstream pipelines. * **Data Sanitization:** Existing overflow values in _real\_time\_events\_details_ were manually reduced to allow transformation pipelines to process records. * **Lock Resolution:** An underlying data table lock on improperly terminated processes was identified and cleared, allowing frozen backfills to resume. * **Backfill & Schedule De-confliction:** Automated scheduled tasks that were running concurrently with the manual backfill scripts were resolved, allowing campaign metrics and integration data backfills to complete successfully. **Recurrence Prevention:** To prevent recurrence and improve resilience, Firstup has committed to the following actions: * **Input Sanitation & Capping:** Implement input restrictions on _time\_spent\_reading_ events from the Employee Experience \(EE\) client side, and on the data ingestion layer to cap values to 1 hour \(3,600,000 ms\) before they enter the data processing pipelines. * **Playbook Expansion:** Update the Engineering Playbook with explicit guidance on identifying/clearing process locks, disabling scheduled jobs during manual backfills, and running backfill scripts. * **Pipeline Resilience & Graceful Error Handling:** Evaluate per-record error handling and modular pipeline refactoring to prevent an isolated invalid record from halting global pipeline processing across all customer programs.