Resolved after 3 minutes, affecting Widget and Chat Integrations with CRMs, on October 14, 2026.
**Summary** On 14th October, our shared background-processing infrastructure reached its configured memory limit. This caused some requests that depended on background job creation or processing to fail or experience delays. **Customer impact** During the incident, some integration requests could not be successfully queued for processing and returned errors. Background workers also encountered memory-related errors. The full impact window is from 21:10 UTC - 22:50 UTC. **Root cause** Our background-processing system uses Redis to store queued jobs and recent processing history. A concentrated workload generated multiple jobs containing unusually large, repeated copies of conversation history. Completed and failed jobs retained these payloads for diagnostic purposes. At the same time, historical document-processing events were retaining large result payloads separately from their job records. This created substantial baseline memory usage. The combination exhausted Redis’s configured memory limit. Redis then rejected operations that required additional memory, affecting other workloads sharing the instance. Our retention controls limited the **number of jobs and events**, but did not sufficiently constrain their **total memory footprint**. Shared capacity also allowed one workload’s memory consumption to affect others. **Mitigation** The Redis memory limit was increased, and subsequent inspection confirmed available memory headroom. This addressed the immediate capacity constraint. We are separately validating processing recovery and reconciling requests whose queue submission failed or could not be confirmed. **Corrective and preventive actions** The following changes are planned to address the underlying causes: * Reduce queued payloads by storing references instead of repeated conversation histories. * Reduce retained job and event-history data, including large document-processing results. * Introduce stronger workload isolation and per-customer resource controls. * Improve alerts for memory growth, oversized payloads, and queue submission failures. * Reconcile affected requests and reprocess them where appropriate, with duplicate-processing safeguards. Our investigation supports rapid accumulation of oversized retained data as the immediate cause; the source of the repeated workload is fully recovered and functiona
Affected components
Update timeline
- investigating Oct 14, 2026, 11:29 PM UTC
We are currently investigating this issue.
- investigating Oct 14, 2026, 11:30 PM UTC
We identified a memory-capacity issue affecting integration processing. Some requests encountered errors or processing delays from 9:10PM UTC - 10:50PM UTC on 14th October, 2026
- identified Oct 14, 2026, 11:30 PM UTC
We identified a memory-capacity issue affecting integration processing. Processing tasks and historical results retained more data than our configured capacity could accommodate.
- monitoring Oct 14, 2026, 11:32 PM UTC
We have increased capacity and confirmed available memory headroom. We are monitoring processing health and reviewing affected requests to determine whether reprocessing is required.
- resolved Oct 14, 2026, 11:32 PM UTC
This incident has been resolved.
- postmortem Oct 14, 2026, 11:46 PM UTC
**Summary** On 14th October, our shared background-processing infrastructure reached its configured memory limit. This caused some requests that depended on background job creation or processing to fail or experience delays. **Customer impact** During the incident, some integration requests could not be successfully queued for processing and returned errors. Background workers also encountered memory-related errors. The full impact window is from 21:10 UTC - 22:50 UTC. **Root cause** Our background-processing system uses Redis to store queued jobs and recent processing history. A concentrated workload generated multiple jobs containing unusually large, repeated copies of conversation history. Completed and failed jobs retained these payloads for diagnostic purposes. At the same time, historical document-processing events were retaining large result payloads separately from their job records. This created substantial baseline memory usage. The combination exhausted Redis’s configured memory limit. Redis then rejected operations that required additional memory, affecting other workloads sharing the instance. Our retention controls limited the **number of jobs and events**, but did not sufficiently constrain their **total memory footprint**. Shared capacity also allowed one workload’s memory consumption to affect others. **Mitigation** The Redis memory limit was increased, and subsequent inspection confirmed available memory headroom. This addressed the immediate capacity constraint. We are separately validating processing recovery and reconciling requests whose queue submission failed or could not be confirmed. **Corrective and preventive actions** The following changes are planned to address the underlying causes: * Reduce queued payloads by storing references instead of repeated conversation histories. * Reduce retained job and event-history data, including large document-processing results. * Introduce stronger workload isolation and per-customer resource controls. * Improve alerts for memory growth, oversized payloads, and queue submission failures. * Reconcile affected requests and reprocess them where appropriate, with duplicate-processing safeguards. Our investigation supports rapid accumulation of oversized retained data as the immediate cause; the source of the repeated workload is fully recovered and functiona
Your product runs on more than Fini AI
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services