[Europe, Japan, Australia] - [Orchestrator] - [Delays for ingestion of robot logs and search or display of the logs]

Update timeline

1. investigating Oct 06, 2026, 05:50 PM UTC

   We are investigating reports of an outage impacting [impacted functionality or user symptom] for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

2. identified Oct 06, 2026, 06:02 PM UTC

   We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

3. identified Oct 06, 2026, 06:28 PM UTC

   We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

4. monitoring Oct 06, 2026, 06:43 PM UTC

   We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

5. resolved Oct 06, 2026, 06:44 PM UTC

   We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

6. postmortem Oct 07, 2026, 11:08 AM UTC

   ## Customer impact On , between approximately 15:45 UTC and 18:45 UTC, customers hosted in the Europe, Australia and Japan regions were unable to see recent robot logs in Orchestrator. Logs written during the impact window did not appear in the Jobs and Logs views or through the Robot Logs API, and logs older than approximately 03:00 UTC that day were available only intermittently. The impact window differed per region: - Australia: approximately 15:45 UTC to 18:35 UTC - Europe: approximately 16:35 UTC to 18:00 UTC - Japan: approximately 16:45 UTC to 18:30 UTC Job execution, queues, triggers and all other Orchestrator functionality were not affected. No robot logs were lost. Logs produced during the impact window continued to be collected and became visible again once the issue was mitigated. ## Root cause We were performing a scheduled upgrade of the Elasticsearch service that stores Orchestrator robot logs. The upgrade uses a blue/green approach: a new cluster is created alongside the existing one, the data is restored into it from a backup, the pipeline that ingests new logs is started on it, and only then is traffic switched over. In the affected regions, the traffic switch was approved by the engineer running the upgrade before the data restore and the ingestion catch-up had completed, which is a deviation from our upgrade procedure. This was a human error. As a result, Orchestrator was reading robot logs from a cluster that did not yet contain recent data, while new logs continued to be collected but were not yet indexed on the cluster serving requests. The error was not caught because our deployment tooling relies on the operator to verify data completeness and does not enforce the order between the data restore and the traffic switch. ## Detection Our synthetic monitoring, which continuously runs test jobs and verifies that their logs are returned, detected missing logs in the affected regions and raised alerts. Our engineering team correlated the alerts with the in-progress upgrade, declared an incident and published a status page notification. ## Response Our engineering team reverted the traffic to the previous Elasticsearch clusters in all three regions, which immediately restored access to all robot logs, including those produced during the impact window. We confirmed recovery through synthetic monitoring in every affected region and continued monitoring before marking the incident resolved at approximately 18:45 UTC. The upgrade of the new clusters continues in the background. Traffic will be moved to them only after the data restore is complete and ingestion is confirmed to be up to date. ## Follow up To prevent this from happening again, we are: - Changing the deployment pipeline so that the traffic switch cannot run until the data restore and log ingestion on the new cluster are verified as complete. - Adding an automated rollback step to the pipeline, so traffic can be returned to the previous cluster within minutes if a problem is detected after a switch.