Robot allocation changes not reflected in Orchestrator

Update timeline

1. identified May 26, 2026, 01:30 PM UTC

   We identified an issue where robot allocations made at the tenant level were not being reflected in Orchestrator. The root cause was traced to messages throttling caused by exceeded storage quotas. Stale duplicate subscriptions with no active consumers had accumulated unread messages, exhausting the storage limit and blocking new message delivery.

2. monitoring May 26, 2026, 01:46 PM UTC

   The stale subscriptions have been cleaned up and message delivery has been restored. Robot allocation changes are now propagating to Orchestrator as expected. We are actively monitoring the system to ensure stability and confirm no further throttling occurs.

3. resolved May 26, 2026, 01:51 PM UTC

   The issue has been fully resolved and service is back to operational.

4. postmortem Jun 03, 2026, 02:30 PM UTC

   ## Customer impact Between May 26, 2026 at 11:05 AM UTC and May 26, 2026 at 1:46 PM UTC, a subset of customers experienced delays or failures when robot allocation changes made at the tenant level were reflected in Orchestrator. Customers could see a discrepancy between the number of robots shown in the tenant allocation panel and the number available in Orchestrator. The impact was global and affected Orchestrator robot allocation propagation across all regions. The service remained partially operational, but allocation updates did not reliably synchronize. ## Root cause The root cause was storage quota exhaustion in the messaging infrastructure used to notify Orchestrator about licensing and robot allocation changes. Duplicate messaging subscriptions that had no active consumers accumulated unread messages over time. Because message expiration on these subscriptions was configured to retain messages indefinitely, the accumulated messages exhausted the available storage quota. This triggered throttling that blocked delivery of new allocation-change messages to Orchestrator. Because Orchestrator depends on those messages to receive tenant-level allocation updates, some allocation changes were not reflected until the unused subscriptions were removed and message delivery resumed. ## Detection The incident was detected via a customer report at 11:05 AM UTC describing tenant-level robot allocation changes not appearing in Orchestrator. We do have synthetic checks in place for this flow, but because allocation propagation was succeeding intermittently rather than failing outright, the checks did not consistently fail and therefore did not raise an alert. As a result, detection relied entirely on the inbound customer signal. From the initial customer report at 11:05 AM UTC to the point at which engineering documented impact and opened incident at 1:15 PM UTC, roughly **2 hours and 10 minutes** elapsed while the symptom was being reproduced, scoped, and routed to the correct service owners. ## Response At 1:15 PM UTC, our engineering team documented the impact as tenant-level robot allocations not consistently appearing in Orchestrator. By 1:17 PM UTC, analysis identified that notification messages to Orchestrator were blocked by throttling caused by exceeded messaging storage quotas. At 1:30 PM UTC, our public status page was updated to reflect that the issue had been identified. Mitigation initially focused on draining the unused subscriptions by reducing message retention settings and attempting to purge accumulated messages. Neither approach cleared the backlog in production. The effective mitigation was to delete the unused duplicate subscriptions that had high unread message counts, while keeping the active subscriptions in place. After the unused subscriptions were removed, a test allocation change was confirmed to appear correctly in Orchestrator during the response call, and the messaging storage usage dropped from full capacity to a healthy level. At 1:46 PM UTC, our public status was moved to monitoring, with message delivery restored and robot allocation changes propagating as expected. The incident was marked resolved at 1:51 PM UTC. ## Follow up 1. Strengthen alerting on messaging storage quotas and per-subscription backlogs, and tune our synthetic checks to detect intermittent allocation-propagation failures rather than only full outages. 2. Clean up remaining duplicate messaging subscriptions that are not actively consumed, so they cannot accumulate unread messages and exhaust storage again. 3. Review why reducing message retention and purging messages did not clear the backlog in production, and use those findings to define a safer recovery procedure for similar events. 4. Run a bulk synchronization from our licensing system to propagate all current allocations to Orchestrator, so impacted customers do not need to re-allocate manually.