Census had 15 outages in the last 2 years totaling 27h 26m of downtime — averaging 0.6 incidents per month.
There were 15 Census outages since May 28, 2025 totaling 27h 26m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating the issue.
The incident has been resolved. Issue determined to be caused due to a configuration change. Configuration change has been reverted.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Census is currently having issues running Syncs and rendering the Website. Engineers have identified the issue and are remediating.
We have identified and triaged the affected system and systems are coming back up now. Web is operational. We will see Sync delays as we scale back up.
We are still seeing degraded performance and in some cases failing syncs. We are continuing efforts to mitigate as the system repairs.
Syncs are currently not operational. Scheduled syncs should still be queuing up but they won't start running at this time. We will provide an update once workers come back up.
Syncs are now operational. Users may be experiencing issues loading Sync Configuration and Sync Run details on the web.
The web is now operating normally, along with Sync Runs. We are monitoring the system for overall health.
This incident has been resolved. Syncs have been running normally as of approximately 1:00pm EST, and the web has been running at normal latencies as of 2:00pm EST. The issue was caused by an indexing change on our side that impacted our ability to run syncs reliably. Recovery was delayed because we needed to rebuild the previous indexes. To avoid any risk of data loss, we chose a safer recovery approach that allowed the system to remain temporarily degraded while repairs were being completed. To restore syncs as quickly as possible, we prioritized sync execution and applied targeted fixes to improve performance. During this time, access to the UI was temporarily down.
Census is experiencing an outage on web, and we are experiencing degraded sync run activity. We have identified the cause and have applied the fix. We are monitoring system recovery.
Census Web has recovered and we are analyzing Sync Runs for impact.
After investigation, a very small number of Sync Runs may have failed with an inability to connect to our Redis instance. The majority of syncs that ran between 1:19 pm and 1:28 pm EST will have experienced a few minutes of delay, and recovered automatically.
The issue has been identified and a fix is being implemented.
The error impacting some customer syncs has now been resolved and syncs are recovering. The syncs impacted were runs with no new data changes to process. This happened due to a recent update that removed the part of our system that correctly handled "no change" scenarios. As a result, syncs without updates failed instead of completing successfully. Syncs with actual data changes were unaffected. We've rolled back the change, and all syncs are now running normally.
The error impacting some customer syncs has now been resolved and syncs are recovering. The syncs impacted were runs with no new data changes to process. This happened due to a recent update that removed the part of our system that correctly handled "no change" scenarios. As a result, syncs without updates failed instead of completing successfully. Syncs with actual data changes were unaffected. We've rolled back the change, and all syncs are now running normally.
This incident has been resolved.
We are currently investigating minor delays affecting some Census Sync job types in the US region. Most syncs are completing successfully, though some may take longer than usual.
We’ve identified the cause of the minor delays and implemented a fix. Sync performance returning back to normal. We’re monitoring to ensure continued stability.
Delays have been resolved. Syncs are running normally again.
In our US region, we are seeing increased latency in non-sync background tasks like metadata refreshes and connection tests due to issues scaling up worker load to match demand, as a result of the continued downstream provider outage. We have identified the issue and are now mitigating the impact.
Non-sync background tasks are running normally. Email notifications are still delayed. We have detected syncs that are currently stuck in working state due to issues earlier in the day. We are working to unblock them.
We are seeing increases in Sync runtimes as we continue to experience issues adding new workers to our pool. We are continuing work to mitigate this issue.
We're now able to appropriately scale workers to meet demands. We are through our main backlogs (including mail sends), and Sync run latencies have improved substantially. We are still continuing to see transient errors throughout Sync runs, mainly dealing with issues hitting our Redis instances, but the frequency of these errors is continuing to decrease.
Our systems show sustained recovery and remain stable. We will continue to monitor for any further issues.
Our system has been operating normally for the last few hours, and the aforementioned downstream provider has marked their incident as resolved. We are marking the sync engine as operational. Please contact support if you believe you are continuing to see issues.
Census experienced major service degradation for running Syncs and on the web for users viewing Sync details for a period of about 3 hours, from 2:46 am EST until 5:30 am EST, due to a large-scale downstream provider outage. All systems are now operational.
We have detected intermittent page load failures on the Census UI due to DNS. Our team is investigating the root cause. Sync runs are unaffected.
This incident has been resolved and page loads are now fully operational. During a system upgrade it was determined that our DNS resolver returned empty values for app.getcensus.com for a brief point - our team will continue investigating the root cause of why this occurred.
We have identified an incident that has caused Census to stop sending all emails. This includes emails related to sync alerts as well as password reset emails and verification emails for new account signups. We are working to remediate the issue. The Census UI and Sync Engine are not otherwise affected
We have updated the our configurations for sending emails and we are currently verifying the fix.
This incident has been resolved.
There are no known problems in production but we are monitoring this AWS incident presently as it has impacted some things in our staging environment. https://health.aws.amazon.com/health/status
AWS has marked this as resolved.
We are aware of the current GCP outage and we are monitoring its impact on Census syncs to and from Google and Google Cloud Resources. Census does not run any critical components on GCP but we do use GCP resources, including Cloud Storage, to facilitate interaction with BigQuery and other Google Cloud resources.
Internal monitoring indicates that syncs to and from BigQuery started experiencing increased failure rates at around 18:05 UTC. The rate of failures is now decreasing but has not returned to usual levels.
Our monitoring systems continue to see a sustained error rate when communicating with Google BigQuery. We will monitor and report when the issue is resolved from Census' perspective. Google Cloud has provided additional information on their status page, located here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Census systems remain unaffected as GCP comes back online. We are now seeing the majority of our GCP hosted customers have successful syncs.
Some syncs started failing with the following error at 12:18pm (EST) "ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column sync_configurations.new_diff_timestamp_expression_id does not exist" This was due to a brief issue during a schema update. The problem has been resolved, and we are monitoring.
This incident has been resolved.
The Census sync list UI is not loading for some customers. We have identified a faulty deployment and we are reverting it. The sync engine is not affected.
A fix for this issue is currently being deployed and will be in place in the next 15-30 minutes
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
During a window of time from 9:58 am EST until 10:32 am EST, the Census web UI, and a portion of the Census Public Management API experienced an outage. Sync Runs were unaffected and operated normally.