Census Outage History
Census is up right nowPingoru has recorded 50 incidents for Census: 13 in the last 12 months and 5 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 0 of 720 (0%) |
| May 2026 | 1 | 6 of 744 (1%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 1 | 3 of 744 (0%) |
| August 2026 | 1 | 1 of 744 (0%) |
| September 2026 | 3 | 7 of 720 (1%) |
| October 2026 | 1 | 3 of 586 (1%) |
Incidents, newest first
Minor March 26, 2025 Mirror Syncs Failing with SQL error 21m Sync Engine
Started Mar 26, 2025, 04:03 AM UTC· resolved Mar 26, 2025, 04:25 AM UTC
- investigating Mar 26, 2025, 04:03 AM UTC
We have identified a bug that is causing "mirror" syncs to fail with an error in the SQL generated by Census; the error message varies by data warehouse, but relates to a missing column. We are rolling back the incorrect deployment. Note that this is unrelated to the scheduled maintenance for Census Store, which has not yet started and will be postponed until this issue is resolved.
- monitoring Mar 26, 2025, 04:15 AM UTC
We have reverted to the previous working version of the system and sync success rates have returned to normal levels. We are investigating why our integration tests were insufficient to prevent this issue.
- resolved Mar 26, 2025, 04:25 AM UTC
This incident has been resolved.
Minor March 21, 2025 New SaaS Dataset, CSV Dataset, and ER Dataset creation unavailable for some customers 22h 37m Census Sync Management UI
Started Mar 21, 2025, 05:57 PM UTC· resolved Mar 22, 2025, 04:35 PM UTC
- identified Mar 21, 2025, 05:57 PM UTC
We have identified the cause and are working on a resolution.
- monitoring Mar 21, 2025, 07:46 PM UTC
We have applied a fix and functionality has been restored. We are continuing to monitor.
- resolved Mar 22, 2025, 04:35 PM UTC
This incident has been resolved.
Major February 5, 2025 Some syncs failing with "Missing corresponding dataset file for prefix: record_deletes.0" 51m Sync Engine
Started Feb 05, 2025, 10:38 PM UTC· resolved Feb 05, 2025, 11:30 PM UTC
- identified Feb 05, 2025, 10:38 PM UTC
A recent code change introduced a bug resulting in some syncs failing with the message "Missing corresponding dataset file for prefix: record_deletes.0". We have identified the problematic change, and a revert is in progress.
- monitoring Feb 05, 2025, 10:56 PM UTC
The revert has deployed. We are monitoring to ensure that the errors subside.
- resolved Feb 05, 2025, 11:30 PM UTC
No errors have been seen for over 30 minutes. We are declaring the incident resolved.
Minor January 30, 2025 Degraded sync performance 12m Sync Engine
Started Jan 30, 2025, 09:44 PM UTC· resolved Jan 30, 2025, 09:57 PM UTC
- investigating Jan 30, 2025, 09:44 PM UTC
Our monitoring has notified us that some syncs are performing more slowly than usual - we are investigating
- resolved Jan 30, 2025, 09:57 PM UTC
This incident has been resolved.
Critical January 30, 2025 IP Address Change for Database Requests 31m Sync Engine
Started Jan 30, 2025, 02:59 AM UTC· resolved Jan 30, 2025, 03:31 AM UTC
- investigating Jan 30, 2025, 02:59 AM UTC
Due to a networking misconfiguration, requests originating from Census's database access service are originating from IPs that we have not previously advertised. If your database is configured to reject requests that are not from Census' IP addresses, syncs will start failing. We are investigating the root cause of the issue and it is our intent to restore service from the original IP addresses; we do not expect customers to need to reconfigure their databases at this time. More updates will be posted here.
- identified Jan 30, 2025, 03:12 AM UTC
We have identified the root cause and are applying a fix to the networking configuration.
- monitoring Jan 30, 2025, 03:24 AM UTC
We have rolled back the service to a previous configuration and we are monitoring our system to ensure database / warehouse connectivity returns to normal levels.
- resolved Jan 30, 2025, 03:31 AM UTC
This incident has been resolved.
Major January 25, 2025 Many syncs failing for 30 minute period 16m Sync Engine
Started Jan 25, 2025, 02:11 AM UTC· resolved Jan 25, 2025, 02:28 AM UTC
- monitoring Jan 25, 2025, 02:11 AM UTC
A misconfiguration of our sync engine resulted in many syncs failing from approximately 2025-01-25 01:08 to 01:38 UTC. We have identified the issue and deployed a fix.
- resolved Jan 25, 2025, 02:28 AM UTC
We have not seen any further failure and are declaring the incident resolved.
Major January 7, 2025 Census unavailable. 503 Error page from the Web. 35m Census Sync Management UI, Sync Engine
Started Jan 07, 2025, 08:53 PM UTC· resolved Jan 07, 2025, 09:28 PM UTC
- investigating Jan 07, 2025, 08:53 PM UTC
We are investigating the issue and will update ASAP.
- investigating Jan 07, 2025, 08:56 PM UTC
The root cause has been identified and we are working to resolve.
- identified Jan 07, 2025, 09:10 PM UTC
The Census UI is now back up. Our sync engine is in the process of coming back up.
- monitoring Jan 07, 2025, 09:18 PM UTC
All services are operational. Any queued syncs are starting up now.
- resolved Jan 07, 2025, 09:28 PM UTC
Everything is now 100% operational. The problem came from an upgrade of our infrastructure that inadvertently removed services. We are planning now to make sure this dependency problem won't be at risk in future upgrades.
Minor December 17, 2024 Slowness and some requests failing 2m Census Sync Management UI, Sync Engine
Started Dec 17, 2024, 09:18 PM UTC· resolved Dec 17, 2024, 09:21 PM UTC
- investigating Dec 17, 2024, 09:18 PM UTC
We are currently investigating this issue.
- resolved Dec 17, 2024, 09:21 PM UTC
The outage was due to a db migration that took longer than was anticipated. Should be resolved now and will monitor for further slowness.
Minor November 21, 2024 Connection tests failing 3h 6m Census Sync Management UI
Started Nov 21, 2024, 02:51 AM UTC· resolved Nov 21, 2024, 05:58 AM UTC
- investigating Nov 21, 2024, 02:51 AM UTC
We are currently investigating an issue where Source & Destination connection tests result in 401 errors.
- monitoring Nov 21, 2024, 05:55 AM UTC
We've confirmed the issue only affects a few connection tests, and the issue has stopped as of the last hour from deploying an old version of our application. We will continue to monitor for connection tests failing with 401 errors.
- resolved Nov 21, 2024, 05:58 AM UTC
We've confirmed the issue only affects a few connection tests, and the issue has stopped as of the last hour from deploying an old version of our application. We will continue to monitor for connection tests failing with 401 errors.
Minor November 15, 2024 Sync Completion Rate Slowed 26m Sync Engine
Started Nov 15, 2024, 04:06 AM UTC· resolved Nov 15, 2024, 04:33 AM UTC
- investigating Nov 15, 2024, 04:06 AM UTC
Syncs are completing at slower than usual rates and we are investigating the issue
- identified Nov 15, 2024, 04:16 AM UTC
We have identified stuck sync workers and we are replacing these containers with new capacity
- monitoring Nov 15, 2024, 04:24 AM UTC
We have rolled back a feature that we believe caused the slowdown and added extra capacity to compensate for the slowdown. Sync completion rates have returned to normal levels and any slowdown or queueing should be resolved in the next 30 minutes.
- resolved Nov 15, 2024, 04:33 AM UTC
This incident has been resolved.