Harness experienced a minor incident on June 4, 2026 affecting Streaming service, lasting 2h 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 04, 2026, 02:33 PM UTC
Instability in streaming connections could cause a delay in propagating updates. All SDKs are designed to poll against the CDN on streaming disconnects and instability scenarios. This could also cause some noise in the logs as SDKs try to reconnect to streaming. But updates are unaffected.
- monitoring Jun 04, 2026, 03:13 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 04, 2026, 04:38 PM UTC
This incident has been resolved.
- postmortem Jul 01, 2026, 11:19 PM UTC
### Summary On June 4, 2026, customers using Harness FME SDKs experienced degraded streaming performance. SDK connections repeatedly disconnected and fell back to polling mode, resulting in delayed flag propagation and elevated log noise. Flag evaluation itself was unaffected — all SDKs are designed to poll against the CDN during streaming disconnects. ### Impact * **Region affected:** us-east-1 \(other regions — EU, APAC — remained healthy\) * **Customer impact:** SDK streaming connections intermittently dropped, causing SDKs to fall back to polling. Flag evaluation continued uninterrupted; customers experienced delayed flag propagation ### Root Cause The underlying cause was memory exhaustion in the backend messaging cluster in the us-east-1 region, triggered by a combination of an ongoing service rollout and a traffic spike during the same . Other regions remained fully healthy throughout. ### Remediation Actions Taken 1. Service was rolled back to the previously stable version. 2. Increased capacity to handle the surge of connections. ### Preventive Measures / Next Steps To prevent such issues happening again, Harness is working on several optimizations: * **Enhance** deployments to do validation before broader rollout to catch capacity pressure early. * **Improve** connection handling during deployments or more conservative deployment pacing during high-load periods. * **Improved SDK behavior on reconnect**: Connection recycles inherently generate a resync storm; further investigation ongoing into reducing the cascading request impact from mass reconnects