Blue Canvas incident

Service outage caused by major Amazon AWS incident

Critical Resolved View vendor source →

Blue Canvas experienced a critical incident on November 25, 2020 affecting Blue Canvas APIs and Metadata Sync (Inbound) and 1 more component, lasting 19h 47m. The incident has been resolved; the full update timeline is below.

Started
Nov 25, 2020, 03:21 PM UTC
Resolved
Nov 26, 2020, 11:09 AM UTC
Duration
19h 47m
Detected by Pingoru
Nov 25, 2020, 03:21 PM UTC

Affected components

Blue Canvas APIsMetadata Sync (Inbound)Metadata Deploy (Outbound)

Update timeline

  1. investigating Nov 25, 2020, 02:24 PM UTC

    Blue Canvas is currently experiencing a service outage. We know this is frustrating and we’re working to resolve this as soon as possible. Next update in 30 minutes.

  2. investigating Nov 25, 2020, 03:08 PM UTC

    We continue to investigate this issue, which may be caused by an outage in AWS US-EAST-1. Our team is currently working to restore the service.

  3. identified Nov 25, 2020, 03:21 PM UTC

    Blue Canvas is currently affected by a major AWS outage affecting multiple infrastructure services in the us-east-1 region. Our team will restore access as soon as AWS is back online. More info: https://status.aws.amazon.com/

  4. identified Nov 25, 2020, 08:07 PM UTC

    The issues with AWS are still ongoing. We are monitoring the situation and will restore service as fast as possible. Please contact Blue Canvas Support if you require urgent access.

  5. identified Nov 25, 2020, 10:31 PM UTC

    We have restored read-only access to Deployment Requests. You can list and view deployments, and leave comments and reviews. Clicking the "Quick Deploy" button will currently not work. We are currently not importing changes from Salesforce. Our next step is to restore read-write access, beginning with a small subset of customers.

  6. monitoring Nov 25, 2020, 11:02 PM UTC

    We are in the final stages of recovery for Blue Canvas and customers should be seeing significant improvement. We expect to be fully recovered within 30 minutes.

  7. monitoring Nov 25, 2020, 11:20 PM UTC

    We have implemented a mitigation to problems caused by the AWS outage. Blue Canvas is now operating normally. Please reach out if you still notice any issues or delays.

  8. resolved Nov 26, 2020, 11:09 AM UTC

    Blue Canvas was impacted by a major outage at AWS that affected a large number of websites. We implemented a mitigation to the impact this had on ECS and Auto-Scaling to restore service. Metadata changes were imported from Salesforce retroactively and the system is now operation normally.