Census incident

Sync Completion Rate Slowed

Minor Resolved View vendor source →

Census experienced a minor incident on November 15, 2024 affecting Sync Engine, lasting 26m. The incident has been resolved; the full update timeline is below.

Started
Nov 15, 2024, 04:06 AM UTC
Resolved
Nov 15, 2024, 04:33 AM UTC
Duration
26m
Detected by Pingoru
Nov 15, 2024, 04:06 AM UTC

Affected components

Sync Engine

Update timeline

  1. investigating Nov 15, 2024, 04:06 AM UTC

    Syncs are completing at slower than usual rates and we are investigating the issue

  2. identified Nov 15, 2024, 04:16 AM UTC

    We have identified stuck sync workers and we are replacing these containers with new capacity

  3. monitoring Nov 15, 2024, 04:24 AM UTC

    We have rolled back a feature that we believe caused the slowdown and added extra capacity to compensate for the slowdown. Sync completion rates have returned to normal levels and any slowdown or queueing should be resolved in the next 30 minutes.

  4. resolved Nov 15, 2024, 04:33 AM UTC

    This incident has been resolved.