Rainforest QA incident

Poor performance for AI related operations

Minor Resolved View vendor source →

Rainforest QA experienced a minor incident on June 7, 2024 affecting Visual Editor and Automation test execution, lasting 2h 35m. The incident has been resolved; the full update timeline is below.

Started
Jun 07, 2024, 12:11 PM UTC
Resolved
Jun 07, 2024, 02:46 PM UTC
Duration
2h 35m
Detected by Pingoru
Jun 07, 2024, 12:11 PM UTC

Affected components

Visual EditorAutomation test execution

Update timeline

  1. investigating Jun 07, 2024, 12:11 PM UTC

    Operations involving AI are taking some time to complete. We're investigating the issue.

  2. monitoring Jun 07, 2024, 12:57 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Jun 07, 2024, 02:46 PM UTC

    This incident has been resolved.

  4. postmortem Jun 11, 2024, 01:16 PM UTC

    We realized that our provider for AI infrastructure was having a soft outage which was causing some requests to take a very long time \(10 minutes\) to error. We worked around this by decreasing the timeout for the requests so that they would error faster and we could then retry them.