pingoru pingoru Coalesce had 26 outages in the last 2 years totaling 516h 9m of downtime, averaging 1.1 incidents per month.
There were 26 Coalesce outages since January 14, 2025 totaling 516h 9m of downtime. Each is summarised below. Incident details, duration, and resolution information.
Transform users are currently unable to create scheduled jobs. We have identified the issue and working on a resolution. This only effects the creation of new scheduled jobs, previously created jobs are continuing to execute.
This issue has been resolved. Please reach out to our Support team if you are experiencing any issues.
Customers are experiencing OAuth authentication issues with Snowflake. We have identified the issue and are pushing out a fix.
We have deployed a fix for anyone experiencing the following error when using OAuth with Snowflake: "error":"Unable to process OAuth callback! error=RPError: iss mismatch, expected snowflake, got: https://example.snowflakecomputing.com/oauth","errorDetail":"" This change introduces case sensitivity for the configured Snowflake URL. If you continue to receive the above error and have a mixed-case URL, a workaround is to change the URL to all lowercase.
This issue has been resolved.
Some of our customers in our APAC region are seeing jobs not completing. We have identified the root cause and will be releasing a fix shortly.
A fix has been deployed and we will monitor for any issues. Further details when fully resolved.
This incident is resolved. This incident was caused by a recent update that lacked backwards compatibility with prior versions of the coa CLI. The resolution was to follow up with another update restoring the backwards compatibility.
We have identified the likely cause of a performance issue with Transform that is effecting overall operations. We will be pushing out a change here shortly to address this and will update this incident when complete.
We have pushed out the change that should resolve this issue. Will monitor and close this out with more details when complete.
We increased resources to our database cluster to better handle customer workloads. This change was rolled out today at approximately 12:15PM Pacific and all operations have returned to normal. We are adding additional monitoring & alerting to improve our response times if this situation occurs in the future.
We are currently experiencing issues establishing connections to GitHub repositories due to a widespread GitHub service disruption. This impacts features within Coalesce that rely on Git repository connectivity, including the Job Scheduler. Our systems are operational, but actions that require communication with GitHub may fail until GitHub restores full service. We are monitoring GitHub’s status page for updates and will post more information as it becomes available. GitHub Status Page: https://www.githubstatus.com/
Github is reporting this as resolved, we are seeing our Github related services working as expected.
We are investigating the impact of a current Github outage that is effecting our Job Scheduler's ability to execute scheduled jobs.
The Github outage is recovering and Coalesce scheduled jobs have resumed processing.
This incident is resolved
We are currently investigating an incident with Catalog functionality.
We have identified the cause of this outage and are working towards a resolution.
This incident has been resolved. A report will be published shortly.
We are aware of an issue impacting all US regions whereby scheduled jobs are not running. We have identified and we are working to triage the issue. Updates will be posted here shortly on status.
We've implemented a fix and are seeing systems return to normal. We are closely monitoring and will provide additional updates shortly.
All issues have been resolved and we are marking this incident as resolved. Please reach out to Coalesce support if you continue to be impacted in any way.
We are currently experiencing service degradation with the Coalesce Job Scheduler. Some scheduled jobs may be delayed or may fail to start. There is no impact on jobs triggered manually, via the API, or via the CLI—those continue to run as expected. Our engineering team is actively investigating the issue. We will provide further updates as more information becomes available.
We have implemented a fix to the Transform Job Scheduler. Scheduled jobs from this point forward should execute successfully as scheduled. If you had any time-sensitive jobs scheduled between 01:00 and 16:30 UTC on 11/26, please review their executions in the Coalesce Deploy window to confirm they ran as expected or to identify any that may require manual intervention. We will continue to monitor the system to ensure continued execution on schedule.
This incident has been resolved.
Please note that we are being impacted by the large AWS us-east-1 issues as detailed on their status page, linked below. We are continuing to monitor the situation and will provide updates as they become available. https://health.aws.amazon.com/health/status
We are seeing additional impacts across all Coalesce Transform regions, due to issues with some of our upstream vendors that we depend here at Coalesce, whom are also directly impacted by the AWS outage. We will continue to monitor and update when there is news to share.
Our upstream vendors and AWS have resolved their issues and all operations have returned to normal for Coalesce Transform. We are therefore closing this incident. Thank you for your understanding and patience during this time.
This incident has been resolved.
This incident has been resolved.
We are aware of intermittent issues and are investigating. We will provide updates as we progress in our investigation.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
All systems continue to be operating normally. We are currently working on identifying the root cause of the issue and will update this incident when determined.
We are continuing to monitor this incident and will provide updates throughout the day.
This incident is resolved and no longer affecting our users and operations. Please reach out to Coalesce Support if you continue to be impacted.
We are investigating an issue where the Coalesce product is not properly fetching saved credentials from secure storage, resulting in failed operations. We will provide the next update by 9:00 PM eastern.
The issue has been identified and a fix is being implemented. We will provide the next update by 9:30 PM eastern.
We are continuing to work on a fix for this issue. We will provide the next update by 10:30 PM eastern.
A fix has been implemented and we are monitoring the results. We will provide the next update by 10:45 PM eastern.
This incident has been resolved.
There appears to be an issue with the Google Firebase service which we use for authentication, causing errors during user login. We are investigating and will continue to provide updates.
We are also observing authentication errors for our Scheduler and API.
There is now an ongoing outage listed on Google Firebase’s authentication service status page: https://status.firebase.google.com/ We expect normal operation once Google resolves their incident.
Updating this incident to Major outage
Starting to see services recover. We still expect to see intermittent errors as services come back online.
Google has resolved their issues and service has returned to normal, we are therefore closing this incident. Please contact Coalesce support directly if there are any concerns.
We have identified an issue impacting Coalesce’s package services, specifically preventing the installation of new packages into workspaces. Our team is actively working on a fix to restore full functionality. We will continue to provide updates as progress is made.
A fix for the issue impacting the Coalesce package services has been identified and our team is working to thoroughly test the software patch. Once all tests have been passed the updated version of the application will be deployed to our production platform and an update will be posted here.
We have deployed an update that corrects the issue impacting the Coalesce package service. Customers are now able to install new packages into workspaces. Please reach out to Support for any assistance.
We are investigating errors received when attempting to commit to git. We will update here as the investigation progresses with any additional information that may be uncovered.
We've identified the issue as only impacting Coalesce clients that utilize GitHub as their git provider. A fix has been created and is currently being tested internally. Once testing is successful a new version of the Coalesce application will be deployed to all production clusters and release notes will be publicly available.
Testing of the software fix is proceeding as expected. Additional updates will be posted here when we deploy the software fix to our production infrastructure.
A fix for this issue has passed testing successfully and was deployed to all Coalesce production clusters. Please refresh any browser sessions to realize the fix that is included in the latest version Coalesce v7.18.2. We continue to monitor this situation for approximately one (1) hour. If our logging remains clear we will mark this incident as resolved.
We're confirming this issue is resolved and has not reoccured after we released the latest version of the Coalesce application.
We're currently monitoring an issue where some users are experiencing 404 error messages when authenticating to Snowflake via OAuth in the Coalesce user interface. Your data and configurations are safe and not affected by this authentication issue. To quickly resolve this on your end: 1. Clear your browser cache: - Chrome: Settings > Privacy and Security > Clear browsing data - Firefox: Options > Privacy & Security > Cookies and Site Data > Clear Data - Edge: Settings > Privacy, search, and services > Clear browsing data - Safari: Preferences > Privacy > Manage Website Data > Remove All 2. Close and restart your browser completely 3. Log back into Coalesce and attempt to reconnect to Snowflake This workaround has resolved the issue for our affected users. If you continue experiencing problems, please contact our Customer Support Team.
The 404 errors are no longer occurring. Please reach out to Support if you experience any issues.
From 3:39 PM to 10:20 PM Eastern, we experienced an issue with our job scheduler. During this period, scheduled jobs failed to start, and notification emails for these failures were not sent. We recommend reviewing the status of any jobs scheduled during this timeframe and manually running any missed jobs if necessary. As of 10:20 PM Eastern, the scheduler issue has been resolved, and jobs will now run as scheduled. We apologize for any inconvenience this may have caused and appreciate your patience and understanding.
We have identified an issue related to our recent release where some static content is being served from a testing domain. This may impact clients with security policies that restrict traffic to specific domain names. Our team is actively working on deploying a fix to resolve this as quickly as possible. We sincerely regret any inconvenience this may cause and appreciate your patience. Temporary Workaround: Allowing content to be served from the domain coalescestage.io may help mitigate the issue while we are working towards a fix. We will provide further updates as we progress toward a resolution. If you have any questions or concerns, please reach out to our support team. Thank you for your understanding. — Coalesce Team
A fix was developed, tested successfully, and deployed to production. The incident is now resolved.
Coalesce is currently experiencing an authentication issue that is impacting API requests. Our team is actively investigating and working to resolve this degradation. We will provide updates as more information becomes available.
The authentication issues impacting API requests have been resolved. We identified the root cause, deployed a fix, and are actively monitoring the system to ensure stability. All services have returned to normal operation. Thank you for your patience during this disruption.