Coalesce Outage History

The hotfix has been delivered in version 7.33.2

The Github outage is recovering and Coalesce scheduled jobs have resumed processing.

The service is online and we are monitoring for any issues.

A fix has been implemented and we are monitoring the results.

We are no longer observing job timeouts at this time. Our team continues to monitor the system closely to ensure stability. We will provide further updates if anything changes.

Jobs are no longer timing out and the Scheduler is back to operational. We are going to continue to monitor over the next 12-24 hours and provide an update tomorrow with more details on the cause of the service interruption.

Between 2-5 AM Pacific Time US we discovered jobs were again backing up and not processing as expected. Engineering has developed a patch to resolve this issue we are targeting to be released by 12 PM Pacific. We will continue to monitor and resolve any issues with job processing and update this incident when the patch has been shipped.

We have released a new version 7.29.4 that mitigates the performance issues our customers are experiencing. We expect it to reduce customer impact from this issue. Any customers using the coa CLI are recommended to upgrade. We will continue to update this issue as we make progress towards the root cause.

Clients can expect to continue to experience issues with delays and timeouts. We have an additional reliability optimization we have developed that is targeted for release within the next 2 hours or less. Expect our next update here when available.

Clients may still see issues with Deploy and Refresh operations which we are continuing to monitor. We have released an additional update, version 7.29.5, that includes follow-on improvements to the reliability of our Refresh and Deploy operations. This update is included in version 7.29.5 of our coa CLI and is a recommended upgrade to all customers. We are continuing to monitor and provide updates as this incident progresses.

The update we released yesterday has resolved the ongoing Deploy and Refresh failures from our clients. We are continuing to monitor and will provide details on the root cause of the issue as soon as available. A small number of our clients are seeing delays running Deploy operations that are unrelated to the ongoing issue this week. We have identified a likely root cause and if confirmed and resolved will transition this issue to Operational.

We are no longer seeing general or widespread issues with our operations and will continue to monitor this incident over the weekend as we work towards a final solution and root cause analysis.

Clients are reporting errors running refresh and deploy operations. We are investigating and will provide updates.

We identified a resource that needed to be restarted and the DEADLINE_EXCEEDED errors clients were seeing should cease. This was unrelated to the overall incident we are tracking here and work will continue to resolving fully.

Starting at approximately 10PM Pacific time Tuesday March 3rd a single resource in our Scheduler API resource pool degraded in performance and did not recover. This caused a performance impact across the product due to the reliance on the Scheduler as our core API service. At 9AM Pacific time Wednesday March 4th, this resource was restarted and operations returned to normal. We are continuing to root cause the reason we experienced a similar scenario on the AM of March 3rd. We have added additional monitoring and alerting for this specific behavior to reduce any impact if it re-occurs until we deliver a full resolution. Updates to follow with a full timeline and root cause of this entire incident.

Incident Update — March 5, 2026 On the afternoon of March 4th, we confirmed the core root cause of this incident. We have been monitoring our infrastructure continuously since then and are not seeing ongoing impact to our clients. Root Cause: This incident was caused by a series of compounding issues, but the primary root cause is a performance flaw in a third-party SQL parsing library used throughout our platform. We identified the problematic behavior early in our investigation, but because SQL parsing is a centralized function critical to Deploy, Refresh, and API operations, we could not simply disable it. A fix required coordination with the third-party vendor, who delivered us a resolution this morning. What We Did While Awaiting the Vendor Fix: Rather than wait, we invested heavily in hardening the platform against the impact of this flaw: - Reliability improvements: Added timeout and retry mechanisms to prevent jobs from hanging indefinitely on unresponsive connections - Resource isolation: Separated heavy SQL parsing workloads onto dedicated infrastructure so they no longer block critical scheduler operations - Performance fixes: Identified and resolved a code-level regression that was amplifying the parsing cost per request - Monitoring & alerting: Deployed new observability tooling and health checks to detect and automatically remediate the application when a degraded state is detected Today's Release: Today we released version 7.30.6, which includes additional reliability and performance improvements that further reduce the impact of the underlying parsing flaw. We will continue to monitor closely and will ship the vendor's fix as soon as it is available. Next Steps: - We are preparing a detailed Incident Report that will be available upon request, including a full timeline and additional details. - We are testing an updated fix from our vendor and expect to release an update early next week, pending our quality checks. We appreciate your patience and collaboration during this ongoing impact to your operations.