Coalesce Outage History

Coalesce Transform for customers in our North America region experienced a service degradation starting on Wednesday April 29th with the release of 7.33 that is affecting job execution. - Root cause: A change in 7.33 made our node selector matching case-insensitive, which in a rare circumstance, made it possible for newly created jobs enter a recursive job-launch loop. - Current state: We have applied a concurrency limit that prevents runaway job launches and are working on a hotfix to resolve the underlying issue. - Fix inbound ASAP: Hotfix reverting selector matching to case-sensitive is approved and in our release pipeline now. We will update this incident once resolved.

We have identified the cause of this outage and are working towards a resolution.

We are aware of an issue impacting all US regions whereby scheduled jobs are not running. We have identified and we are working to triage the issue. Updates will be posted here shortly on status.

Please note that we are being impacted by the large AWS us-east-1 issues as detailed on their status page, linked below. We are continuing to monitor the situation and will provide updates as they become available. https://health.aws.amazon.com/health/status

The issue has been identified and a fix is being implemented. We will provide the next update by 9:30 PM eastern.

We are continuing to work on a fix for this issue. We will provide the next update by 10:30 PM eastern.

We have identified an issue impacting Coalesce’s package services, specifically preventing the installation of new packages into workspaces. Our team is actively working on a fix to restore full functionality. We will continue to provide updates as progress is made.

A fix for the issue impacting the Coalesce package services has been identified and our team is working to thoroughly test the software patch. Once all tests have been passed the updated version of the application will be deployed to our production platform and an update will be posted here.

We've identified the issue as only impacting Coalesce clients that utilize GitHub as their git provider. A fix has been created and is currently being tested internally. Once testing is successful a new version of the Coalesce application will be deployed to all production clusters and release notes will be publicly available.

Testing of the software fix is proceeding as expected. Additional updates will be posted here when we deploy the software fix to our production infrastructure.

We have identified an issue related to our recent release where some static content is being served from a testing domain. This may impact clients with security policies that restrict traffic to specific domain names. Our team is actively working on deploying a fix to resolve this as quickly as possible. We sincerely regret any inconvenience this may cause and appreciate your patience. Temporary Workaround: Allowing content to be served from the domain coalescestage.io may help mitigate the issue while we are working towards a fix. We will provide further updates as we progress toward a resolution. If you have any questions or concerns, please reach out to our support team. Thank you for your understanding. — Coalesce Team

Coalesce is currently experiencing an authentication issue that is impacting API requests. Our team is actively investigating and working to resolve this degradation. We will provide updates as more information becomes available.

We are seeing GitHub outages that are impacting certain functionality within the Coalesce application: - All job scheduling is not operable - All customers using Github for version control within the Coalesce application will see additional issues impacting the ability to run jobs. We are monitoring the situation and will update this page when appropriate.