Census had 40 outages in the last 2 years totaling 70h 23m of downtime — averaging 1.6 incidents per month.
There were 40 Census outages since June 3, 2024 totaling 70h 23m of downtime. Each is summarised below — incident details, duration, and resolution information.
Syncs to Microsoft Advertising temporarily experienced failures with an error message like "OAuth2::Error: invalid_client" from approximately 05/21/2026 17:00 UTC - 05/21/20206 23:19 UTC. This error was the result of the Activations Microsoft Advertising app configuration and it has now been resolved. Any impacted syncs will recover on their next scheduled run.
We are currently investigating the issue.
The incident has been resolved. Issue determined to be caused due to a configuration change. Configuration change has been reverted.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Census is currently having issues running Syncs and rendering the Website. Engineers have identified the issue and are remediating.
We have identified and triaged the affected system and systems are coming back up now. Web is operational. We will see Sync delays as we scale back up.
We are still seeing degraded performance and in some cases failing syncs. We are continuing efforts to mitigate as the system repairs.
Syncs are currently not operational. Scheduled syncs should still be queuing up but they won't start running at this time. We will provide an update once workers come back up.
Syncs are now operational. Users may be experiencing issues loading Sync Configuration and Sync Run details on the web.
The web is now operating normally, along with Sync Runs. We are monitoring the system for overall health.
This incident has been resolved. Syncs have been running normally as of approximately 1:00pm EST, and the web has been running at normal latencies as of 2:00pm EST. The issue was caused by an indexing change on our side that impacted our ability to run syncs reliably. Recovery was delayed because we needed to rebuild the previous indexes. To avoid any risk of data loss, we chose a safer recovery approach that allowed the system to remain temporarily degraded while repairs were being completed. To restore syncs as quickly as possible, we prioritized sync execution and applied targeted fixes to improve performance. During this time, access to the UI was temporarily down.
Census is experiencing an outage on web, and we are experiencing degraded sync run activity. We have identified the cause and have applied the fix. We are monitoring system recovery.
Census Web has recovered and we are analyzing Sync Runs for impact.
After investigation, a very small number of Sync Runs may have failed with an inability to connect to our Redis instance. The majority of syncs that ran between 1:19 pm and 1:28 pm EST will have experienced a few minutes of delay, and recovered automatically.
The issue has been identified and a fix is being implemented.
The error impacting some customer syncs has now been resolved and syncs are recovering. The syncs impacted were runs with no new data changes to process. This happened due to a recent update that removed the part of our system that correctly handled "no change" scenarios. As a result, syncs without updates failed instead of completing successfully. Syncs with actual data changes were unaffected. We've rolled back the change, and all syncs are now running normally.
The error impacting some customer syncs has now been resolved and syncs are recovering. The syncs impacted were runs with no new data changes to process. This happened due to a recent update that removed the part of our system that correctly handled "no change" scenarios. As a result, syncs without updates failed instead of completing successfully. Syncs with actual data changes were unaffected. We've rolled back the change, and all syncs are now running normally.
This incident has been resolved.
We are currently investigating minor delays affecting some Census Sync job types in the US region. Most syncs are completing successfully, though some may take longer than usual.
We’ve identified the cause of the minor delays and implemented a fix. Sync performance returning back to normal. We’re monitoring to ensure continued stability.
Delays have been resolved. Syncs are running normally again.
In our US region, we are seeing increased latency in non-sync background tasks like metadata refreshes and connection tests due to issues scaling up worker load to match demand, as a result of the continued downstream provider outage. We have identified the issue and are now mitigating the impact.
Non-sync background tasks are running normally. Email notifications are still delayed. We have detected syncs that are currently stuck in working state due to issues earlier in the day. We are working to unblock them.
We are seeing increases in Sync runtimes as we continue to experience issues adding new workers to our pool. We are continuing work to mitigate this issue.
We're now able to appropriately scale workers to meet demands. We are through our main backlogs (including mail sends), and Sync run latencies have improved substantially. We are still continuing to see transient errors throughout Sync runs, mainly dealing with issues hitting our Redis instances, but the frequency of these errors is continuing to decrease.
Our systems show sustained recovery and remain stable. We will continue to monitor for any further issues.
Our system has been operating normally for the last few hours, and the aforementioned downstream provider has marked their incident as resolved. We are marking the sync engine as operational. Please contact support if you believe you are continuing to see issues.
Census experienced major service degradation for running Syncs and on the web for users viewing Sync details for a period of about 3 hours, from 2:46 am EST until 5:30 am EST, due to a large-scale downstream provider outage. All systems are now operational.
We have detected intermittent page load failures on the Census UI due to DNS. Our team is investigating the root cause. Sync runs are unaffected.
This incident has been resolved and page loads are now fully operational. During a system upgrade it was determined that our DNS resolver returned empty values for app.getcensus.com for a brief point - our team will continue investigating the root cause of why this occurred.
We have identified an incident that has caused Census to stop sending all emails. This includes emails related to sync alerts as well as password reset emails and verification emails for new account signups. We are working to remediate the issue. The Census UI and Sync Engine are not otherwise affected
We have updated the our configurations for sending emails and we are currently verifying the fix.
This incident has been resolved.
There are no known problems in production but we are monitoring this AWS incident presently as it has impacted some things in our staging environment. https://health.aws.amazon.com/health/status
AWS has marked this as resolved.
We are aware of the current GCP outage and we are monitoring its impact on Census syncs to and from Google and Google Cloud Resources. Census does not run any critical components on GCP but we do use GCP resources, including Cloud Storage, to facilitate interaction with BigQuery and other Google Cloud resources.
Internal monitoring indicates that syncs to and from BigQuery started experiencing increased failure rates at around 18:05 UTC. The rate of failures is now decreasing but has not returned to usual levels.
Our monitoring systems continue to see a sustained error rate when communicating with Google BigQuery. We will monitor and report when the issue is resolved from Census' perspective. Google Cloud has provided additional information on their status page, located here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Census systems remain unaffected as GCP comes back online. We are now seeing the majority of our GCP hosted customers have successful syncs.
Some syncs started failing with the following error at 12:18pm (EST) "ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column sync_configurations.new_diff_timestamp_expression_id does not exist" This was due to a brief issue during a schema update. The problem has been resolved, and we are monitoring.
This incident has been resolved.
The Census sync list UI is not loading for some customers. We have identified a faulty deployment and we are reverting it. The sync engine is not affected.
A fix for this issue is currently being deployed and will be in place in the next 15-30 minutes
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
During a window of time from 9:58 am EST until 10:32 am EST, the Census web UI, and a portion of the Census Public Management API experienced an outage. Sync Runs were unaffected and operated normally.
A recent code change introduced a bug resulting in some syncs failing with the message "T.cast: Expected type T::Struct, got type TrueClass". This impacted sync runs where there were no incremental changes identified and the source connection utilizes the Advanced Sync Engine. We identified the problematic change, and have reverted. Syncs are now running successfully.
We've identified an issue that caused background jobs to be delayed. The issue does not impact running syncs.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating issue with some syncs failing with mismatch errors.
We experienced a performance issue from ~3:00-3:15pm Pacific that may have caused some syncs to fail with a message about "mismatch counts". We believe we have resolved the problem, but syncs that encountered the issue may not report an error until sometime after the fact, so errors may continue to trickle in. If you encounter this error, please attempt to rerun your sync. If a rerun does not succeed, please contact support.
This issue is resolved.
We are continuing to monitor for any further issues.
This incident has been resolved.
We have identified a bug that is causing "mirror" syncs to fail with an error in the SQL generated by Census; the error message varies by data warehouse, but relates to a missing column. We are rolling back the incorrect deployment. Note that this is unrelated to the scheduled maintenance for Census Store, which has not yet started and will be postponed until this issue is resolved.
We have reverted to the previous working version of the system and sync success rates have returned to normal levels. We are investigating why our integration tests were insufficient to prevent this issue.
This incident has been resolved.
We have identified the cause and are working on a resolution.
We have applied a fix and functionality has been restored. We are continuing to monitor.
This incident has been resolved.
A recent code change introduced a bug resulting in some syncs failing with the message "Missing corresponding dataset file for prefix: record_deletes.0". We have identified the problematic change, and a revert is in progress.
The revert has deployed. We are monitoring to ensure that the errors subside.
No errors have been seen for over 30 minutes. We are declaring the incident resolved.
Our monitoring has notified us that some syncs are performing more slowly than usual - we are investigating
This incident has been resolved.
Due to a networking misconfiguration, requests originating from Census's database access service are originating from IPs that we have not previously advertised. If your database is configured to reject requests that are not from Census' IP addresses, syncs will start failing. We are investigating the root cause of the issue and it is our intent to restore service from the original IP addresses; we do not expect customers to need to reconfigure their databases at this time. More updates will be posted here.
We have identified the root cause and are applying a fix to the networking configuration.
We have rolled back the service to a previous configuration and we are monitoring our system to ensure database / warehouse connectivity returns to normal levels.
This incident has been resolved.
A misconfiguration of our sync engine resulted in many syncs failing from approximately 2025-01-25 01:08 to 01:38 UTC. We have identified the issue and deployed a fix.
We have not seen any further failure and are declaring the incident resolved.