Blue Canvas incident

Queue Overloaded

Minor Resolved View vendor source →

Blue Canvas experienced a minor incident on June 11, 2026 affecting Metadata Deploy (Outbound), lasting 5h 12m. The incident has been resolved; the full update timeline is below.

Started
Jun 11, 2026, 03:37 PM UTC
Resolved
Jun 11, 2026, 08:49 PM UTC
Duration
5h 12m
Detected by Pingoru
Jun 11, 2026, 03:37 PM UTC

Affected components

Metadata Deploy (Outbound)

Update timeline

  1. identified Jun 11, 2026, 03:37 PM UTC

    A queue that processes background jobs has become overloaded, and is affecting the creation of deployment requests. We're currently working to resolve this issue.

  2. monitoring Jun 11, 2026, 04:23 PM UTC

    We identified the root cause. One of our queued jobs, scheduled to run every 10 seconds, began taking several minutes to complete. These long-running jobs accumulated and consumed all available worker capacity. We've deployed two fixes: - Concurrency limit, only one instance of this job can run at a time, preventing pile-up. - Rewritten algorithm, the job now resolves in ~1 second instead of minutes. We're continuing to monitor.

  3. resolved Jun 11, 2026, 08:49 PM UTC

    This incident has been resolved.