Census experienced a minor incident on October 20, 2025 affecting Sync Engine, lasting 10h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 20, 2025, 01:43 PM UTC
In our US region, we are seeing increased latency in non-sync background tasks like metadata refreshes and connection tests due to issues scaling up worker load to match demand, as a result of the continued downstream provider outage. We have identified the issue and are now mitigating the impact.
- identified Oct 20, 2025, 04:07 PM UTC
Non-sync background tasks are running normally. Email notifications are still delayed. We have detected syncs that are currently stuck in working state due to issues earlier in the day. We are working to unblock them.
- identified Oct 20, 2025, 05:54 PM UTC
We are seeing increases in Sync runtimes as we continue to experience issues adding new workers to our pool. We are continuing work to mitigate this issue.
- monitoring Oct 20, 2025, 06:52 PM UTC
We're now able to appropriately scale workers to meet demands. We are through our main backlogs (including mail sends), and Sync run latencies have improved substantially. We are still continuing to see transient errors throughout Sync runs, mainly dealing with issues hitting our Redis instances, but the frequency of these errors is continuing to decrease.
- monitoring Oct 20, 2025, 08:51 PM UTC
Our systems show sustained recovery and remain stable. We will continue to monitor for any further issues.
- resolved Oct 21, 2025, 12:26 AM UTC
Our system has been operating normally for the last few hours, and the aforementioned downstream provider has marked their incident as resolved. We are marking the sync engine as operational. Please contact support if you believe you are continuing to see issues.