Resolved after 2 hours 4 minutes, affecting Streaming service, on June 4, 2026.
### Summary On June 4, 2026, customers using Harness FME SDKs experienced degraded streaming performance. SDK connections repeatedly disconnected and fell back to polling mode, resulting in delayed flag propagation and elevated log noise. Flag evaluation itself was unaffected — all SDKs are designed to poll against the CDN during streaming disconnects. ### Impact * **Region affected:** us-east-1 \(other regions — EU, APAC — remained healthy\) * **Customer impact:** SDK streaming connections intermittently dropped, causing SDKs to fall back to polling. Flag evaluation continued uninterrupted; customers experienced delayed flag propagation ### Root Cause The underlying cause was memory exhaustion in the backend messaging cluster in the us-east-1 region, triggered by a combination of an ongoing service rollout and a traffic spike during the same . Other regions remained fully healthy throughout. ### Remediation Actions Taken 1. Service was rolled back to the previously stable version. 2. Increased capacity to handle the surge of connections. ### Preventive Measures / Next Steps To prevent such issues happening again, Harness is working on several optimizations: * **Enhance** deployments to do validation before broader rollout to catch capacity pressure early. * **Improve** connection handling during deployments or more conservative deployment pacing during high-load periods. * **Improved SDK behavior on reconnect**: Connection recycles inherently generate a resync storm; further investigation ongoing into reducing the cascading request impact from mass reconnects
Affected components
Update timeline
- identified Jun 04, 2026, 02:33 PM UTC
Instability in streaming connections could cause a delay in propagating updates. All SDKs are designed to poll against the CDN on streaming disconnects and instability scenarios. This could also cause some noise in the logs as SDKs try to reconnect to streaming. But updates are unaffected.
- monitoring Jun 04, 2026, 03:13 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 04, 2026, 04:38 PM UTC
This incident has been resolved.
- postmortem Jul 01, 2026, 11:19 PM UTC
### Summary On June 4, 2026, customers using Harness FME SDKs experienced degraded streaming performance. SDK connections repeatedly disconnected and fell back to polling mode, resulting in delayed flag propagation and elevated log noise. Flag evaluation itself was unaffected — all SDKs are designed to poll against the CDN during streaming disconnects. ### Impact * **Region affected:** us-east-1 \(other regions — EU, APAC — remained healthy\) * **Customer impact:** SDK streaming connections intermittently dropped, causing SDKs to fall back to polling. Flag evaluation continued uninterrupted; customers experienced delayed flag propagation ### Root Cause The underlying cause was memory exhaustion in the backend messaging cluster in the us-east-1 region, triggered by a combination of an ongoing service rollout and a traffic spike during the same . Other regions remained fully healthy throughout. ### Remediation Actions Taken 1. Service was rolled back to the previously stable version. 2. Increased capacity to handle the surge of connections. ### Preventive Measures / Next Steps To prevent such issues happening again, Harness is working on several optimizations: * **Enhance** deployments to do validation before broader rollout to catch capacity pressure early. * **Improve** connection handling during deployments or more conservative deployment pacing during high-load periods. * **Improved SDK behavior on reconnect**: Connection recycles inherently generate a resync storm; further investigation ongoing into reducing the cascading request impact from mass reconnects
Your product runs on more than Harness
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services