Census Outage History

Census is up right now

Census had 40 outages in the last 2 years totaling 40h 48m of downtime — averaging 1.6 incidents per month.

There were 40 Census outages since June 3, 2024 totaling 40h 48m of downtime. Each is summarised below — incident details, duration, and resolution information.

Source: https://status.getcensus.com

Major January 7, 2025

Census unavailable. 503 Error page from the Web.

Detected by Pingoru
Jan 07, 2025, 08:53 PM UTC
Resolved
Jan 07, 2025, 09:28 PM UTC
Duration
35m
Affected: Census Sync Management UISync Engine
Timeline · 5 updates
  1. investigating Jan 07, 2025, 08:53 PM UTC

    We are investigating the issue and will update ASAP.

  2. investigating Jan 07, 2025, 08:56 PM UTC

    The root cause has been identified and we are working to resolve.

  3. identified Jan 07, 2025, 09:10 PM UTC

    The Census UI is now back up. Our sync engine is in the process of coming back up.

  4. monitoring Jan 07, 2025, 09:18 PM UTC

    All services are operational. Any queued syncs are starting up now.

  5. resolved Jan 07, 2025, 09:28 PM UTC

    Everything is now 100% operational. The problem came from an upgrade of our infrastructure that inadvertently removed services. We are planning now to make sure this dependency problem won't be at risk in future upgrades.

Read the full incident report →

Minor December 17, 2024

Slowness and some requests failing

Detected by Pingoru
Dec 17, 2024, 09:18 PM UTC
Resolved
Dec 17, 2024, 09:21 PM UTC
Duration
2m
Affected: Census Sync Management UISync Engine
Timeline · 2 updates
  1. investigating Dec 17, 2024, 09:18 PM UTC

    We are currently investigating this issue.

  2. resolved Dec 17, 2024, 09:21 PM UTC

    The outage was due to a db migration that took longer than was anticipated. Should be resolved now and will monitor for further slowness.

Read the full incident report →

Minor November 21, 2024

Connection tests failing

Detected by Pingoru
Nov 21, 2024, 02:51 AM UTC
Resolved
Nov 21, 2024, 05:58 AM UTC
Duration
3h 6m
Affected: Census Sync Management UI
Timeline · 3 updates
  1. investigating Nov 21, 2024, 02:51 AM UTC

    We are currently investigating an issue where Source & Destination connection tests result in 401 errors.

  2. monitoring Nov 21, 2024, 05:55 AM UTC

    We've confirmed the issue only affects a few connection tests, and the issue has stopped as of the last hour from deploying an old version of our application. We will continue to monitor for connection tests failing with 401 errors.

  3. resolved Nov 21, 2024, 05:58 AM UTC

    We've confirmed the issue only affects a few connection tests, and the issue has stopped as of the last hour from deploying an old version of our application. We will continue to monitor for connection tests failing with 401 errors.

Read the full incident report →

Minor November 15, 2024

Sync Completion Rate Slowed

Detected by Pingoru
Nov 15, 2024, 04:06 AM UTC
Resolved
Nov 15, 2024, 04:33 AM UTC
Duration
26m
Affected: Sync Engine
Timeline · 4 updates
  1. investigating Nov 15, 2024, 04:06 AM UTC

    Syncs are completing at slower than usual rates and we are investigating the issue

  2. identified Nov 15, 2024, 04:16 AM UTC

    We have identified stuck sync workers and we are replacing these containers with new capacity

  3. monitoring Nov 15, 2024, 04:24 AM UTC

    We have rolled back a feature that we believe caused the slowdown and added extra capacity to compensate for the slowdown. Sync completion rates have returned to normal levels and any slowdown or queueing should be resolved in the next 30 minutes.

  4. resolved Nov 15, 2024, 04:33 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor October 2, 2024

Sync performance degraded

Detected by Pingoru
Oct 02, 2024, 06:02 AM UTC
Resolved
Oct 02, 2024, 06:45 AM UTC
Duration
42m
Affected: Sync Engine
Timeline · 3 updates
  1. investigating Oct 02, 2024, 06:02 AM UTC

    We are currently investigating an issue causing slow sync execution. The team has been paged and is currently investigating the cause of the issue. Census management console is unaffected and there are no problems with sync completion at this time. Only the speed of sync execution is currently affected. We will update the status page once the issue is identified.

  2. monitoring Oct 02, 2024, 06:31 AM UTC

    An issue with an earlier deployment was identified as cause of the issue. The issue has been fixed and sync engine has largely recovered. The team is monitoring the recovery and expect it to be resolved shortly.

  3. resolved Oct 02, 2024, 06:45 AM UTC

    The team considers this issue resolved. We will continue to monitor for delayed issues but do not expect any further impact.

Read the full incident report →

Major September 20, 2024

Sync Alerts missing as of last 2 days

Detected by Pingoru
Sep 20, 2024, 08:18 PM UTC
Resolved
Sep 20, 2024, 10:55 PM UTC
Duration
2h 37m
Affected: Sync Engine
Timeline · 3 updates
  1. identified Sep 20, 2024, 08:18 PM UTC

    Sync Alerts were deleted from our database in the last 2 days. The cause has been identified, and we are working on restoring the deleted alerts.

  2. identified Sep 20, 2024, 09:36 PM UTC

    As part of this incident, any sync runs that should have triggered an alert in the last 2 days would not have resulted in an email alert. Once we restore the alert data, any sync runs that fail will trigger an alert, even though the sync may have been failing for the last 2 days. In very rare cases, we may email you at most twice about a failure as we perform the recovery process.

  3. resolved Sep 20, 2024, 10:55 PM UTC

    Sync Alert data has been restored. Sync alerting should work again for any syncs that trigger alerts moving forward.

Read the full incident report →

Major September 11, 2024

Partial Sync Outage

Detected by Pingoru
Sep 11, 2024, 07:00 PM UTC
Resolved
Sep 12, 2024, 12:06 AM UTC
Duration
5h 5m
Affected: Sync Engine
Timeline · 4 updates
  1. identified Sep 11, 2024, 07:00 PM UTC

    Syncs empty unload batches have a chance of failing with either a "IO Error: No files found that match the pattern IGNORED_EMPTY_FILE" or a "Parquet Conversion Error". The issue has been identified and a fix is being released.

  2. monitoring Sep 11, 2024, 07:23 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. investigating Sep 11, 2024, 08:50 PM UTC

    A subset of the previously affected syncs are now hitting "No function matches the given name and argument types" errors. It's primarily limited to advanced sync engine mirror syncs with no deletes. Investigation continues.

  4. resolved Sep 12, 2024, 12:06 AM UTC

    The issue has been resolved.

Read the full incident report →

Minor September 5, 2024

Increased Sync Failure Rate

Detected by Pingoru
Sep 05, 2024, 06:56 PM UTC
Resolved
Sep 05, 2024, 08:00 PM UTC
Duration
1h 4m
Affected: Sync Engine
Timeline · 3 updates
  1. identified Sep 05, 2024, 06:56 PM UTC

    We are seeing instances of Syncs failing with a message indicating "Parquet conversion failed." The problematic system has been identified and we are working to mitigate and resolve the issue as soon as possible.

  2. monitoring Sep 05, 2024, 07:44 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Sep 06, 2024, 01:15 AM UTC

    This incident has been resolved.

Read the full incident report →

Critical August 23, 2024

Sync execution halted

Detected by Pingoru
Aug 23, 2024, 05:41 PM UTC
Resolved
Aug 23, 2024, 07:15 PM UTC
Duration
1h 33m
Affected: Sync Engine
Timeline · 6 updates
  1. identified Aug 23, 2024, 05:41 PM UTC

    Production sync execution has been paused while we investigate an issue

  2. identified Aug 23, 2024, 05:58 PM UTC

    We have identified an issue which could cause records to be synced multiple times to destinations. Since this can affect the correctness of syncs for certain destinations, we have paused all syncs proactively until the issue can be diagnosed and resolved. We are continuing to investigate and are in the act of rolling back to a previous version of the sync engine.

  3. identified Aug 23, 2024, 06:42 PM UTC

    We have determined that the bug does not affect all syncs, nor does it cause records to be sent multiple times - this was a red herring. A high percentage of syncs are still failing, but there is not a correctness issue, so we are reenabling the sync engine globally so that syncs that are not affected by the issue can return to operation.

  4. identified Aug 23, 2024, 06:48 PM UTC

    We have identified the likely root cause and are deploying a fix at this time

  5. monitoring Aug 23, 2024, 07:04 PM UTC

    Sync success rates have returned to expected levels. There is some delay in sync execution as we work through the backlog during the outage period. We are continuing to monitor

  6. resolved Aug 23, 2024, 07:15 PM UTC

    This incident has been resolved.

Read the full incident report →

Major August 16, 2024

Linkedin Syncs Failing

Detected by Pingoru
Aug 16, 2024, 04:57 PM UTC
Resolved
Aug 16, 2024, 06:36 PM UTC
Duration
1h 39m
Affected: Sync Engine
Timeline · 3 updates
  1. identified Aug 16, 2024, 04:57 PM UTC

    Linkedin Syncs are failing as the API version used by Census has been deprecated. The issue has been identified, and we are bumping the API version to unblock syncs.

  2. monitoring Aug 16, 2024, 05:33 PM UTC

    We have fixed the issue. All Linkedin syncs affected by the issue should succeed the next time they run.

  3. resolved Aug 16, 2024, 06:36 PM UTC

    This incident has been resolved.

Read the full incident report →

Notice July 31, 2024

Elevated Error Rates Detected

Detected by Pingoru
Jul 31, 2024, 12:47 AM UTC
Resolved
Jul 31, 2024, 03:23 AM UTC
Duration
2h 36m
Affected: Census Sync Management UISync Engine
Timeline · 3 updates
  1. investigating Jul 31, 2024, 12:47 AM UTC

    We are currently seeing increased error rates in our us region.

  2. monitoring Jul 31, 2024, 02:19 AM UTC

    Mitigation has been deployed to aid in reducing failures from our upstream provider. We have seen a significant reduction in errors and are now monitoring the system.

  3. resolved Jul 31, 2024, 03:23 AM UTC

    This incident has been resolved.

Read the full incident report →

Notice July 18, 2024

[ROLLBACK OF] Modified logic for calculating Failed Records % could result in new Sync Alert emails

Detected by Pingoru
Jul 18, 2024, 10:22 PM UTC
Resolved
Jul 18, 2024, 10:22 PM UTC
Duration
—
Timeline · 1 update
  1. resolved Jul 18, 2024, 10:49 PM UTC

    This change has been rolled back to reduce noisy alerts for existing customers. Any alerts that were opened yesterday due to this change have been closed. Recovery emails will not be sent for the alerts opened yesterday. Failed record alerts will be triggered with the old logic moving forward. ------------Original Update-------------------------- To improve accuracy, we recently updated the calculation that powers the Failed Records % Sync Alert. This one-time correction could result in new alerts to those syncs where the correction applies. The new logic calculates the percent of failed records based on the number of changed records in a sync run, rather than the total number of records in the source.

Read the full incident report →

Minor June 27, 2024

Delays starting scheduled syncs

Detected by Pingoru
Jun 27, 2024, 08:46 PM UTC
Resolved
Jun 28, 2024, 05:44 PM UTC
Duration
20h 57m
Affected: Sync Engine
Timeline · 3 updates
  1. investigating Jun 27, 2024, 08:46 PM UTC

    We are currently experiencing delays of up to 20 minutes starting scheduled syncs due to a downstream vendor dependency. This is especially pronounced for syncs that run at common times (e.g., around hour boundaries). We are working with the vendor to identify and fix the issue.

  2. monitoring Jun 28, 2024, 03:36 AM UTC

    We have identified the issue, which appears to have predominantly impacted syncs scheduled to run around 9am Eastern/6am Pacific. We have put in place a mitigation and will monitor sync runs running that time tomorrow to make sure the issue does not recur.

  3. resolved Jun 28, 2024, 05:44 PM UTC

    We have confirmed that syncs are now running as scheduled.

Read the full incident report →

Major June 22, 2024

Sync runs failing

Detected by Pingoru
Jun 22, 2024, 01:29 PM UTC
Resolved
Jun 22, 2024, 01:43 PM UTC
Duration
14m
Affected: Sync Engine
Timeline · 3 updates
  1. investigating Jun 22, 2024, 01:29 PM UTC

    A subset of sync runs are failing due to an infrastructure issue. We are investigating

  2. monitoring Jun 22, 2024, 01:38 PM UTC

    We have identified a storage issue affecting a core database and applied a fix - error rates are decreasing

  3. resolved Jun 22, 2024, 01:43 PM UTC

    This incident has been resolved.

Read the full incident report →

Major June 3, 2024

Brief downtime for unscheduled database maintenance

Detected by Pingoru
Jun 03, 2024, 07:37 PM UTC
Resolved
Jun 03, 2024, 07:42 PM UTC
Duration
4m
Affected: Census Sync Management UISync Engine
Timeline · 2 updates
  1. identified Jun 03, 2024, 07:37 PM UTC

    We are performing unscheduled emergency maintenance of one of our primary databases. Users may see brief downtime of the Census UI and API. Syncs that fail due to this will be automatically retried

  2. resolved Jun 03, 2024, 07:42 PM UTC

    This incident has been resolved.

Read the full incident report →