Netwrix incident

1Secure: Activity Import Delay

Major Resolved View vendor source →

Netwrix experienced a major incident on April 7, 2025 affecting Activity and Activity, lasting 44d 1h. The incident has been resolved; the full update timeline is below.

Started
Apr 07, 2025, 02:40 PM UTC
Resolved
May 21, 2025, 04:17 PM UTC
Duration
44d 1h
Detected by Pingoru
Apr 07, 2025, 02:40 PM UTC

Affected components

ActivityActivity

Update timeline

  1. identified Mar 26, 2025, 12:08 PM UTC

    We have identified a delay in the import of activity records in the US region. The team have identified the cause and are working on a resolution.

  2. identified Mar 31, 2025, 09:03 AM UTC

    We are continuing to work on a fix for this issue.

  3. identified Apr 07, 2025, 02:40 PM UTC

    We've stabilized the service and restored full operations for live activity data. We identified that the service had dropped activity events and it is now processing these older events from the import queue. We are monitoring this process and anticipate it completing in the next few days. We will provide another update on the progress of importing the older events within 48 hours.

  4. identified Apr 09, 2025, 02:00 PM UTC

    The overall state of activity record imports in both environments has improved. We are continuing to monitor and tune performance as needed to resume normal service.

  5. identified Apr 10, 2025, 02:05 PM UTC

    Historical Activity records have now been imported in the EU instance and normal service has now resumed. We will continue to monitor this environment closely to ensure its stability. The US environment is continuing to improve and we will provide an update as soon as this is complete.

  6. monitoring Apr 11, 2025, 02:21 PM UTC

    Historical Activity records have now been imported in the US instance. Normal service has now resumed in both the EU and US. We will continue to monitor closely.

  7. resolved May 21, 2025, 04:17 PM UTC

    The team have been monitoring the activity feed closely over the past few weeks to validate the resolution of this issue and alerts are being recalculated to include the impacted activity records. Customers are advised to review the alerts dashboard to check for any alerts that require additional action. The team will continue to monitor the feed and are deploying additional automation to improve scaling response times to better meet the highly variable rate of data ingestion.