Robocorp incident

Work Item system failures

Major Resolved View vendor source →

Robocorp experienced a major incident on November 11, 2022 affecting Robots and API, lasting 1h 26m. The incident has been resolved; the full update timeline is below.

Started
Nov 11, 2022, 08:04 AM UTC
Resolved
Nov 11, 2022, 09:31 AM UTC
Duration
1h 26m
Detected by Pingoru
Nov 11, 2022, 08:04 AM UTC

Affected components

RobotsAPI

Update timeline

  1. investigating Nov 11, 2022, 08:04 AM UTC

    We are currently investigating Internal server error responses from the work item system.

  2. resolved Nov 11, 2022, 09:31 AM UTC

    The situation is back to normal. Approximately 10% of requests to the task data subsystem between 6:54 UTC and 7:54 UTC were failing with HTTP status code 500. Impacted step runs may have failed depending on which actions the robot has taken. Run failure notifications have been operational during the time, thus surfacing the issue to robot operators. The failed work items can be retried from the control room. We apologize for the inconvenience.