Blue Canvas incident

Queue Overloaded

Resolved after 5 hours 12 minutes, affecting Metadata Deploy (Outbound), on June 11, 2026.

monitoring Jun 11, 2026, 04:23 PM UTC

We identified the root cause. One of our queued jobs, scheduled to run every 10 seconds, began taking several minutes to complete. These long-running jobs accumulated and consumed all available worker capacity. We've deployed two fixes: - Concurrency limit, only one instance of this job can run at a time, preventing pile-up. - Rewritten algorithm, the job now resolves in ~1 second instead of minutes. We're continuing to monitor.

Started
Jun 11, 2026, 03:37 PM UTC
Resolved
Jun 11, 2026, 08:49 PM UTC
Duration
5h 12m
Detected by Pingoru
Jun 11, 2026, 03:37 PM UTC

Affected components

Metadata Deploy (Outbound)

Update timeline

  1. identified Jun 11, 2026, 03:37 PM UTC

    A queue that processes background jobs has become overloaded, and is affecting the creation of deployment requests. We're currently working to resolve this issue.

  2. monitoring Jun 11, 2026, 04:23 PM UTC

    We identified the root cause. One of our queued jobs, scheduled to run every 10 seconds, began taking several minutes to complete. These long-running jobs accumulated and consumed all available worker capacity. We've deployed two fixes: - Concurrency limit, only one instance of this job can run at a time, preventing pile-up. - Rewritten algorithm, the job now resolves in ~1 second instead of minutes. We're continuing to monitor.

  3. resolved Jun 11, 2026, 08:49 PM UTC

    This incident has been resolved.

Your product runs on more than Blue Canvas

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →