Rubrik incident

Ransomware Monitoring and Investigation - Snapshots processing slower than usual

Minor Resolved View vendor source →

Rubrik experienced a minor incident on September 5, 2023, lasting 2d 20h. The incident has been resolved; the full update timeline is below.

Started
Sep 05, 2023, 10:08 PM UTC
Resolved
Sep 08, 2023, 06:59 PM UTC
Duration
2d 20h
Detected by Pingoru
Sep 05, 2023, 10:08 PM UTC

Update timeline

  1. investigating Sep 05, 2023, 10:08 PM UTC

    We are experiencing increased latency for Ransomware Monitoring and Investigation on Rubrik Security Cloud in all regions. Symptom(s): Anomalies may not appear in the dashboard, and there may be a reduction in the count of analyzed snapshots. Incident Start Time: 08/28/2023 10:00 PM PST Workaround: None at this time. Engineering is actively investigating to resolve this issue.

  2. identified Sep 06, 2023, 12:06 AM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Sep 06, 2023, 06:43 AM UTC

    We are continuing to work on a fix for this issue.

  4. identified Sep 06, 2023, 04:21 PM UTC

    We are continuing to work on a fix for this issue.

  5. identified Sep 06, 2023, 07:54 PM UTC

    We are still continuing to work on a fix for this issue. We will give another update at 19:00 PST.

  6. identified Sep 07, 2023, 12:25 AM UTC

    We believe the change will start rolling out by 09/07/2023 01:00 PST, we will be continuously monitoring the service to evaluate the effectiveness of the mitigation.

  7. identified Sep 07, 2023, 03:03 PM UTC

    The change is finished rolling out, we are continuously monitoring the service to evaluate the effectiveness of the mitigation. We will provide another update at 10:00 PST.

  8. identified Sep 07, 2023, 05:25 PM UTC

    We are working on few more fixes for this issue. We will give another update at 18:00 PST.

  9. monitoring Sep 08, 2023, 12:18 AM UTC

    A fix has been implemented and we are monitoring the results.

  10. monitoring Sep 08, 2023, 06:02 AM UTC

    The issue has been fixed and we are seeing signs of improvement. We are further monitoring and will provide an update.

  11. resolved Sep 08, 2023, 06:59 PM UTC

    The fixes we have implemented have solved the issue. Although some customers may experience slow snapshot processing for the next two to three days.