Rainforest QA incident

VM Availability Issues

Major Resolved View vendor source →

Rainforest QA experienced a major incident on February 20, 2023 affecting Visual Editor and Automation test execution, lasting 10h 33m. The incident has been resolved; the full update timeline is below.

Started
Feb 20, 2023, 02:45 PM UTC
Resolved
Feb 21, 2023, 01:19 AM UTC
Duration
10h 33m
Detected by Pingoru
Feb 20, 2023, 02:45 PM UTC

Affected components

Visual EditorAutomation test execution

Update timeline

  1. investigating Feb 20, 2023, 02:45 PM UTC

    We've detected a problem with VMs not loading in the Visual Editor and are investigating.

  2. identified Feb 20, 2023, 04:27 PM UTC

    We have identified and mitigated the issue. We're still experiencing reduced availability and are working to fully resolve it.

  3. identified Feb 20, 2023, 06:11 PM UTC

    We've resolved the issue that was causing VMs to fail to load in the Visual Editor, we're now working to recover our full capacity. This incident has also affected our capacity to execute tests as part of a run, so runs may be delayed until we've fully recovered.

  4. resolved Feb 21, 2023, 01:19 AM UTC

    We are running at full capacity.