Blue Canvas incident

Errors between Blue Canvas and Salesforce.

Major Resolved View vendor source →

Blue Canvas experienced a major incident on May 10, 2021 affecting Metadata Sync (Inbound) and Metadata Deploy (Outbound) and 1 more component, lasting 15h 13m. The incident has been resolved; the full update timeline is below.

Started
May 10, 2021, 04:09 PM UTC
Resolved
May 11, 2021, 07:22 AM UTC
Duration
15h 13m
Detected by Pingoru
May 10, 2021, 04:09 PM UTC

Affected components

Metadata Sync (Inbound)Metadata Deploy (Outbound)Salesforce APIs

Update timeline

  1. investigating May 10, 2021, 04:09 PM UTC

    Some customers are experiencing errors. Affected customers will see the following error in the deployment logs: FATAL: Could not read '.tenantId' from response. Affected customers are also unable to connect a new organization to Blue Canvas, resulting in "Internal server error"

  2. investigating May 10, 2021, 04:52 PM UTC

    So far we found strong indicators that this happens when interacting with Salesforce sandboxes. Errors started appearing last Saturday during the Summer '21 update deployed by Salesforce. We continue to investigate the situation.

  3. investigating May 10, 2021, 05:46 PM UTC

    We are continuing to investigate this issue.

  4. monitoring May 10, 2021, 05:50 PM UTC

    We identified the issue. With release 52.0 Salesforce is returning http 500 responses for randomly when we request the profile. This api call is necessary for us to login. As a workaround we implemented retry mechanism. This should resolve all cases, however we are still monitoring the situation.

  5. resolved May 11, 2021, 07:22 AM UTC

    Even-tough Salesforce continues to give random errors, we are marking this incident as resolved. Our workaround seems effective in resolving these errors in over 99% of the cases. If you still receive this error manually retrying once will most likely resolve the issue. If not feel free to contact support.