Customer.io incident
Some workspaces experiencing delays in campaign journeys and deliveries in US Data Center
Customer.io experienced a minor incident on November 4, 2025 affecting Data Processing and Message Sending, lasting 1h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 04, 2025, 08:46 PM UTC
We are currently investigating this issue.
- monitoring Nov 04, 2025, 09:12 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 04, 2025, 09:54 PM UTC
This incident has been resolved.
- postmortem Nov 13, 2025, 07:36 PM UTC
# **Incident Summary** On November 4, 2025, between 20:10 UTC and 21:54 UTC, some customers with our US data center may have experienced delays in message sending. This issue did not affect data ingestion and no data was lost. A configuration change caused a failure in processing attribute updates from campaign attribute & webhook actions resulting in campaign journeys stalling. The configuration change was reverted, restoring the attribute update processing and message sending. # **Root Cause** A configuration change to the internal authentication system used for inter-service communication unexpectedly caused our services responsible for attribute processing to fail, leading to a backlog in message sending. # **Resolution and Recovery** Engineers reverted the configuration change at 21:07 UTC, allowing attribute processing to resume successfully. The team monitored the system until confirming recovery at 21:54 UTC. # **Corrective and Preventative Measures** To improve resilience, the team is enhancing monitoring to detect early signs of attribute processing faults and updating alerting to provide clearer failure context and faster remediation. These measures are being tracked and prioritized within our internal development process.