Harness incident
FME - Some customers are experiencing delays in scheduled exports of impressions
Harness experienced a minor incident on July 2, 2026 affecting FME and FME and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 02, 2026, 03:48 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jul 02, 2026, 08:10 PM UTC
We seeing the backlogged items worked through and continuing to monitor results.
- monitoring Jul 02, 2026, 11:31 PM UTC
We are continuing to monitor results while the backlog is worked through.
- resolved Jul 03, 2026, 04:11 PM UTC
This incident has been resolved.
- postmortem Jul 09, 2026, 07:24 PM UTC
**Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).