Census incident

UI slowdown across several areas

Major Resolved View vendor source →

Census experienced a major incident on March 26, 2024 affecting Census Sync Management UI, lasting 53m. The incident has been resolved; the full update timeline is below.

Started
Mar 26, 2024, 09:53 PM UTC
Resolved
Mar 26, 2024, 10:47 PM UTC
Duration
53m
Detected by Pingoru
Mar 26, 2024, 09:53 PM UTC

Affected components

Census Sync Management UI

Update timeline

  1. investigating Mar 26, 2024, 09:53 PM UTC

    We have identified slowdowns in the Census UI associated with several areas: * Model Previews * Metadata Refreshes * Sync Alerts These are related to a slowdown in our job processing engine. At this time, our metrics do not show that sync performance or scheduling is affected but we are investigating these as well.

  2. identified Mar 26, 2024, 10:04 PM UTC

    We have identified a faulty deploy and we are rolling it back

  3. monitoring Mar 26, 2024, 10:27 PM UTC

    A previous version of the job processor has been deployed and these tasks are once again being completed. We are continuing to monitor

  4. resolved Mar 26, 2024, 10:47 PM UTC

    This incident has been resolved.