Blue Canvas Outage History
Blue Canvas is up right nowPingoru has recorded 49 incidents for Blue Canvas: 6 in the last 12 months and 2 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 0 of 720 (0%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 2 | 29 of 720 (4%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 1 | 33 of 744 (4%) |
| September 2026 | 1 | 7 of 720 (1%) |
| October 2026 | 0 | 0 of 509 (0%) |
Incidents, newest first
Major September 16, 2026 Salesforce Outage 7h 6m Metadata Sync (Inbound), Metadata Deploy (Outbound), Salesforce APIs
Started Sep 16, 2026, 09:10 AM UTC· resolved Sep 16, 2026, 04:17 PM UTC
- identified Sep 16, 2026, 09:10 AM UTC
Salesforce is experiencing a platform-wide disruption affecting instances across all regions. As a result, some Blue Canvas syncs and deployments are failing or delayed, and some branches may show as disconnected. Follow Salesforce for updates: https://status.salesforce.com/incidents/20004433
- monitoring Sep 16, 2026, 11:21 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 16, 2026, 04:17 PM UTC
This incident has been resolved.
Minor August 19, 2026 Sync servers crashing 1d 8h Metadata Sync (Inbound), Metadata Deploy (Outbound)
Started Aug 19, 2026, 08:05 AM UTC· resolved Aug 20, 2026, 04:41 PM UTC
- investigating Aug 19, 2026, 08:05 AM UTC
Syncs and deploys are currently not happening. The underlying servers are crashing for various reasons. We're working on pinpointing the exact issue.
- monitoring Aug 19, 2026, 11:19 AM UTC
We have identified the cause of the intermittent delays affecting branch synchronization and related processing over the past day. A fix has been deployed and the affected processing infrastructure has been replaced. All synchronization services are operating normally and we are monitoring closely to confirm the fix holds under full load. Synchronization jobs that were delayed during this period have resumed automatically — no action is required from customers, and no data was lost.
- investigating Aug 19, 2026, 01:06 PM UTC
Unfortunately the system degraded again after ~1 hour.
- monitoring Aug 19, 2026, 01:58 PM UTC
The "ring" that serves syncs and deploys is stable again. This should get the syncs and deploys running as usual. In the meantime we'll keep an eye out and act immediately if something changes.
- monitoring Aug 20, 2026, 09:00 AM UTC
Earlier today we identified an issue where ~9% of branches experienced extra delay on both syncs and deploys. The fixes from yesterday helped containing the issue so the other 91% of branches could continue as normal. We've resolved this issue by scaling up the units that have to do the work. There are currently no known issues, but we are looking for a sustainable solution, which may involve one or more releases. In technical terms: The scaling and self-repair configuration we've used for the last year is no long viable. We've greatly optimized on CPU, to the point where it is no longer a reliable metric for scaling in or out. We've disabled the scaling, and manually put the system on a configuration that works right now, and continue to fine tune it while we work on a permanent and automated solution.
- resolved Aug 20, 2026, 04:41 PM UTC
Systems are operational, no incidents since last update, we don't expect further downtime while we optimize the data center.
Minor June 11, 2026 Queue Overloaded 5h 12m Metadata Deploy (Outbound)
Started Jun 11, 2026, 03:37 PM UTC· resolved Jun 11, 2026, 08:49 PM UTC
- identified Jun 11, 2026, 03:37 PM UTC
A queue that processes background jobs has become overloaded, and is affecting the creation of deployment requests. We're currently working to resolve this issue.
- monitoring Jun 11, 2026, 04:23 PM UTC
We identified the root cause. One of our queued jobs, scheduled to run every 10 seconds, began taking several minutes to complete. These long-running jobs accumulated and consumed all available worker capacity. We've deployed two fixes: - Concurrency limit, only one instance of this job can run at a time, preventing pile-up. - Rewritten algorithm, the job now resolves in ~1 second instead of minutes. We're continuing to monitor.
- resolved Jun 11, 2026, 08:49 PM UTC
This incident has been resolved.
Minor June 9, 2026 All Branches Require Reconnection 1d Metadata Sync (Inbound)
Started Jun 09, 2026, 09:31 AM UTC· resolved Jun 10, 2026, 09:37 AM UTC
- monitoring Jun 09, 2026, 09:31 AM UTC
Due to a recent, Salesforce-mandated security upgrade, all branches need to be reconnected.
- resolved Jun 10, 2026, 09:37 AM UTC
This incident has been resolved.
Critical January 29, 2026 Deployment Requests failing to package 40m Metadata Deploy (Outbound)
Started Jan 29, 2026, 03:57 PM UTC· resolved Jan 29, 2026, 04:38 PM UTC
- investigating Jan 29, 2026, 03:57 PM UTC
Deployment requests are currently failing to package with the message "The read operation timed out". It seems like our authentication requests to Salesforce are being timed out, and not being retried very often. We're currently pushing a fix to retry more frequently while we continue to investigate.
- monitoring Jan 29, 2026, 04:27 PM UTC
We've implemented a fix and Deployment Requests are successfully building once again. Any that are currently marked as failed because the read operation timed out can be safely retried.
- resolved Jan 29, 2026, 04:38 PM UTC
# Postmortem: Salesforce Authentication & Deployment Delays **Date Range:** January 27-29, 2025 **Status:** Resolved ## Summary Between January 27-29, some customers experienced intermittent deployment failures and delays when Blue Canvas attempted to authenticate with Salesforce. The issues have been fully resolved. ## What Happened ### January 27 We made infrastructure improvements to support customers who require IP allowlisting for Salesforce access. All Blue Canvas traffic to Salesforce is now routed through 5 static IP addresses, making it easier for customers to configure their Salesforce security settings. We also improved visibility into authentication issues by adding warnings to deployment operations when they are blocked due to Salesforce authentication errors. ### January 28 While implementing the authentication error warnings, we inadvertently introduced a configuration change that caused temporary errors to persist longer than necessary. When Salesforce experienced brief slowdowns, our system would cache these errors for up to 1 hour, preventing automatic recovery even after Salesforce responded normally again. ### January 29 Salesforce's authentication service experienced intermittent slowdowns, causing some authentication requests to time out. Combined with our caching behavior, this resulted in deployments being marked as failed instead of retrying automatically. ## Impact - Customers experienced intermittent deployment failures with messages like "Failed to get Salesforce token" or authentication timeout errors - Some deployments required manual retry after Salesforce recovered ## Resolution We implemented the following fixes: 1. **Smarter Error Handling:** Our system now distinguishes between temporary issues (like timeouts) and permanent errors (like invalid credentials). Temporary issues trigger automatic retry instead of immediate failure. 2. **Faster Recovery:** Temporary errors are now cached for only 1 minute instead of 1 hour, allowing much faster recovery when Salesforce service is restored. 3. **Improved User Communication:** When Salesforce is slow to respond, you'll now see the message: *"Packaging is taking longer because Salesforce is not responding. We will retry automatically."* instead of an immediate failure. 4. **Automatic Retry:** Deployments affected by temporary Salesforce issues will now retry automatically without requiring manual intervention. ## Questions? If you have any questions or experienced issues during this period that haven't been resolved, please contact our support team.
Major December 22, 2025 Branches not syncing 59m Metadata Sync (Inbound)
Started Dec 22, 2025, 12:07 PM UTC· resolved Dec 22, 2025, 01:06 PM UTC
- identified Dec 22, 2025, 12:07 PM UTC
We have identified the root cause and implemented a fix. However, this fix requires a long-running database migration that is already in progress and running smoothly. Branch syncing is temporarily paused while the migration runs. No data will be lost, but new syncs will not start until the migration completes. We apologize for the disruption and expect normal service to resume shortly. We'll post another update once the migration finishes and syncing is restored.
- resolved Dec 22, 2025, 01:06 PM UTC
This incident has been resolved.
Notice October 21, 2025 AWS Outage 1d
Started Oct 21, 2025, 03:00 AM UTC· resolved Oct 22, 2025, 03:15 AM UTC
- resolved Oct 21, 2025, 08:15 AM UTC
Between 20 Oct 2025, 07:11 UTC and 21 Oct 2025, 03:15 UTC, our services experienced degraded performance and partial unavailability due to a widespread AWS infrastructure outage affecting multiple regions. During this period, users may have encountered failed requests, timeouts, or intermittent service disruptions.
Minor March 11, 2025 Database issues 31m Blue Canvas Web Application, Blue Canvas APIs, Metadata Sync (Inbound), Metadata Deploy (Outbound)
Started Mar 11, 2025, 03:44 PM UTC· resolved Mar 11, 2025, 04:16 PM UTC
- identified Mar 11, 2025, 03:44 PM UTC
We are experiencing issues. The data in the database has been growing at an unusual pace, therefore we have to execute emergency maintanance.
- identified Mar 11, 2025, 03:48 PM UTC
We are continuing to work on a fix for this issue.
- resolved Mar 11, 2025, 04:16 PM UTC
Issue has been resolved
Minor February 13, 2025 Files tab not working in deployment requests 25d 17h Blue Canvas Web Application
Started Feb 13, 2025, 10:01 PM UTC· resolved Mar 11, 2025, 03:41 PM UTC
- investigating Feb 13, 2025, 10:01 PM UTC
The frontend is not updating properly when rendering the file tab. This is due to a significant boost in background jobs on our git server. We are investigating the root cause. Deployments are working, but we understand it is hard to see what is going on without being able to inspect the files.
- monitoring Feb 14, 2025, 11:28 AM UTC
The queues seem to be cleared up and everything is working as expected. We've had to clear some of the queues. We have not yet identified the exact job that was delaying the system. However we've found some great optimizations and working on a long term fix for those meanwhile we are monitoring the situation. If your deployment is still showing as "Nothing to merge from" please use "Discard results and validate again" in the top right dropdown of the deployment request view.
- resolved Mar 11, 2025, 03:41 PM UTC
The incident has been resolved since February 15th, we have been monitoring ever since but the incident has not been properly updated.