Census experienced a minor incident on March 26, 2025 affecting Sync Engine, lasting 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 26, 2025, 04:03 AM UTC
We have identified a bug that is causing "mirror" syncs to fail with an error in the SQL generated by Census; the error message varies by data warehouse, but relates to a missing column. We are rolling back the incorrect deployment. Note that this is unrelated to the scheduled maintenance for Census Store, which has not yet started and will be postponed until this issue is resolved.
- monitoring Mar 26, 2025, 04:15 AM UTC
We have reverted to the previous working version of the system and sync success rates have returned to normal levels. We are investigating why our integration tests were insufficient to prevent this issue.
- resolved Mar 26, 2025, 04:25 AM UTC
This incident has been resolved.