Qlty Software Outage History

Started Oct 21, 2026, 04:27 AM UTC· resolved Oct 21, 2026, 04:05 PM UTC

1. investigating Oct 21, 2026, 04:27 AM UTC

   We are responding to the recently disclosed eMudhra certificate authority compromise: https://emudhra.we-have-a-problem.com/. An attacker able to intercept network traffic could impersonate trusted services to software that still accepts the affected certificates. We have released initial mitigations for our hosted analysis infrastructure. We are continuing to review and remediate Qlty services, build environments, the CLI, and bundled tooling, including installation and upgrade paths. We recommend that CLI and self-hosted CI users apply OS security updates and keep certificate verification enabled. Some developer tools use separate certificate bundles, so an OS or browser update alone may not be enough. We will publish fixed versions, upgrade instructions, and any cache refresh steps when they are available.

2. resolved Oct 21, 2026, 04:05 PM UTC

   Qlty services are running normally. This is a security issue in certificate trust, not an outage. We have published a security advisory with the status of each Qlty component and steps you can take now: https://github.com/qltysh/qlty/security/advisories/GHSA-f55x-97q4-pfrc. We will keep the advisory up to date as fixes ship, and we will stop posting updates here.

Read the full incident report →

Started Oct 08, 2026, 10:17 PM UTC· resolved Oct 08, 2026, 11:32 PM UTC

1. identified Oct 08, 2026, 10:17 PM UTC

   Since 10:35 AM ET (14:35 UTC) on October 8, 2026, analysis builds fail for projects that use the Semgrep plugin. The build stops while installing Semgrep, before any analysis runs. Projects that do not use Semgrep are not affected. Coverage uploads are not affected. A workaround is available now (below). We are sorry for the disruption. What is happening Semgrep 1.144.0, the version Qlty installs by default, depends on a package called mcp that imports a private function from pydantic. pydantic 2.14.0, published at 10:34 AM ET (14:34 UTC) today, removed that function. Semgrep does not limit which pydantic version it installs, so every fresh install of Semgrep now picks up pydantic 2.14.0 and fails to start. The build log shows: ImportError: cannot import name 'eval_type_backport' from 'pydantic._internal._typing_extra' Semgrep releases 1.137.0 through 1.145.0 share this dependency and fail the same way. Releases 1.146.0 and later install correctly. Who is affected - Qlty Cloud: every analysis build for a project with Semgrep enabled fails while installing tools. Affected pull requests do not get an analysis result. - qlty CLI in CI, or on any machine installing Semgrep for the first time: `qlty check` fails while installing Semgrep. - qlty CLI on a machine where Semgrep was installed before 10:34 AM ET today: not affected. The installed copy keeps working until the tool cache is cleared. Workaround, available now in Semgrep to 1.179.0 in `.qlty/qlty.toml`: [[plugin]] name = "semgrep" version = "1.179.0" If your `qlty.toml` already has a `[[plugin]]` entry for `semgrep`, add or change its `version` line. Commit and push. The next Qlty Cloud build and the next `qlty check` run install 1.179.0, which works with pydantic 2.14.0. If you had already pinned Semgrep to a version between 1.137.0 and 1.145.0, you need to make this change yourself. Our fix changes the default version only and does not override a version you set. Semgrep 1.179.0 is 35 releases newer than 1.144.0. Its rules have changed in that time, so you may see new findings or lose some old ones after the switch. We chose 1.179.0 over a smaller jump because every Semgrep release between 1.146.0 and 1.178.0 depends on packages with open security advisories. What we are doing We are changing the default Semgrep version in the qlty CLI from 1.144.0 to 1.179.0 (https://github.com/qltysh/qlty/pull/2910). We will ship it in the next CLI release and roll it out to Qlty Cloud. After that, projects without an explicit Semgrep version build again with no change on your side. Builds that failed during the incident do not rerun on their own. Push a new commit to get a fresh result.

2. resolved Oct 08, 2026, 11:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 05, 2026, 04:18 PM UTC· resolved Oct 05, 2026, 04:18 PM UTC

1. resolved Oct 05, 2026, 04:18 PM UTC

   On October 5, 2026, between 12:18 and 12:31 PM ET, Qlty stopped completing analysis results for 13 minutes. About 120 builds across roughly 60 workspaces never completed. On the affected pull requests, the Qlty check on GitHub showed "Qlty encountered a timeout error at 15mins" while the build page on Qlty showed the build as passed. Pushing a new commit to an affected pull request produces a fresh, correct result. We are sorry for the confusion this caused. Below is what happened, who was affected, why, and what we are changing. Timeline (US Eastern on October 5, 2026, UTC in parentheses) • 12:17 PM (16:17): A routine data cleanup in one workspace started a large maintenance operation on our main database. • 12:18 PM (16:18): Analysis results stopped completing. Builds kept running and finishing normally, but their results were not written. • 12:30 PM (16:30): Our queue-delay alarm fired. • 12:31 PM (16:31): The operation finished and results started completing again. • 12:33 to 12:50 PM (16:33 to 16:50): The Qlty check on affected pull requests changed from pending to the 15-minute timeout error. • 12:35 PM (16:35): Result processing was back to normal. • About 1:00 PM (17:00): A customer reported a pull request stuck on the timeout error. We advised pushing a new commit. • 4:00 PM (20:00): We confirmed the root cause. Who was affected During the 13-minute window, every analysis result was delayed by up to 12 minutes. Most builds recovered on their own: our queue retries each result three times, and 73 builds completed on a later attempt. 118 builds did not complete after three attempts and were dropped. They span 88 pull requests and 26 default-branch pushes, across about 80 projects in roughly 60 workspaces. For each of these builds: • The Qlty check on GitHub showed the timeout error instead of a pass or fail, and the check linked to our troubleshooting docs rather than to the build. • No pull request comment or review was posted. • The build page on Qlty showed the build as passed, with no indication that its results were missing. • For the default-branch pushes, that commit is missing from the project's quality history until the next push. A smaller group of builds that did complete on a retry recorded some issues more than once, which inflated their issue counts. This affects 42 builds. We are deciding whether to correct these or leave them, since a later push replaces them. If you had a pull request open during this window and the Qlty check still shows the timeout error, push a new commit and the check will run again. Root cause A customer ran a routine cleanup of older data in their project. That is a normal, supported action. Behind it, Qlty removes that project's data from one of our largest tables. Because of how that table is laid out, removing one project's data required reading a large share of the whole table to find the small fraction that matched. That read did not slow the database itself as much as it slowed ClickHouse Keeper, the coordination service that tracks what data each database replica has. Every request to Keeper became about six times slower for the 13 minutes the operation ran. The final step of processing an analysis result, which computes a build's metrics, asks Keeper to confirm the replica has the latest data before the query starts. Under normal conditions that check takes under a second. With Keeper slowed, and about 40 of these steps waiting at once, each check queued behind the others and the wait grew to several minutes. Our 30-second client timeout cut each attempt short, the queue retried three times over about six minutes, and builds whose three attempts all fell inside the 13-minute window were dropped. What we are doing • Making data removal a bounded operation. Removing a project's data will cost in proportion to the data being removed, not the size of the table it lives in, so one workspace's cleanup cannot slow the service for everyone. • Reducing the final processing step's dependence on Keeper. We are cutting the number of coordination checks each metrics query makes, and we are designing a version of that step that does not need them at all. • Alerting sooner. We are adding alerts on the database timeouts that preceded the queue delay, so we see an event like this within a couple of minutes instead of twelve. If you have questions about a specific pull request or build from this window, contact us through qlty.sh/contact/support: http://qlty.sh/contact/support and include the pull request or build link.

Read the full incident report →

Started Aug 17, 2026, 02:29 PM UTC· resolved Aug 17, 2026, 10:35 PM UTC

1. identified Aug 17, 2026, 02:29 PM UTC

   We have identified the source of this incident, GitHub's various APIs are experiencing degrading performance https://www.githubstatus.com/incidents/zkxwbgr0cnmx.

2. resolved Aug 17, 2026, 10:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 12, 2026, 07:15 PM UTC· resolved Aug 12, 2026, 11:12 PM UTC

1. monitoring Aug 12, 2026, 07:15 PM UTC

   We are currently investigating intermittent build failures caused by network issues when downloading plugins from GitHub during analysis runs. Our team is actively identifying the root cause and working on a resolution. We will provide an update as soon as more details or a fix are available.

2. identified Aug 12, 2026, 08:32 PM UTC

   Since approximately 9:30 AM PT, a portion of analysis builds have been failing during the tool-installation step, with errors like "Error installing tool …" in the build log. We've traced this to an upstream disruption affecting downloads of release assets from GitHub — the archives our builds fetch to install linters and language runtimes. The disruption is intermittent: during failure bursts roughly 1 in 8 builds fails, and between bursts builds complete normally. We have escalated it to GitHub. What you can do: re-running a failed build will usually succeed, though a retry may still hit a failure window while the disruption continues. No action is needed for builds that completed successfully. Coverage uploads and processing, qlty.sh: http://qlty.sh, and the API are unaffected. We're monitoring failure rates continuously and will post an update when the disruption clears.

3. identified Aug 12, 2026, 09:57 PM UTC

   We have identified the source of this incident and are working on remediation.

4. resolved Aug 12, 2026, 11:12 PM UTC

   GitHub acknowledged that this was an incident: https://www.githubstatus.com/incidents/lsvy8xsf0gxv and resolved it.

Read the full incident report →

Started Aug 06, 2026, 05:01 PM UTC· resolved Aug 07, 2026, 09:04 AM UTC

1. identified Aug 06, 2026, 05:01 PM UTC

   Coverage using Github OIDC appears to be affected by a Github Actions incident https://www.githubstatus.com/incidents/qcvjkzcs7j74

2. identified Aug 06, 2026, 05:02 PM UTC

   We have identified the source of this incident and are working on remediation.

3. identified Aug 06, 2026, 05:02 PM UTC

   Coverage using Github OIDC appears to be affected by a Github Actions incident https://www.githubstatus.com/incidents/qcvjkzcs7j74

4. identified Aug 06, 2026, 09:57 PM UTC

   There have been multiple impacts due to Github outage https://www.githubstatus.com/incidents/qcvjkzcs7j74 such as coverage action using OIDC breaking and delayed results on analysis due to webhooks being throttled.

5. resolved Aug 07, 2026, 09:04 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 16, 2026, 11:00 PM UTC· resolved Jul 17, 2026, 12:15 AM UTC

1. identified Jul 16, 2026, 11:00 PM UTC

   Github API requests are throwing 503, Github has declared an incident and is investigating. https://www.githubstatus.com/incidents/gxycch3076xk

2. monitoring Jul 17, 2026, 12:09 AM UTC

   Github has remediated the source of the incident and are monitoring.

3. resolved Jul 17, 2026, 12:15 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 16, 2026, 02:44 PM UTC· resolved Jun 16, 2026, 06:00 PM UTC

1. investigating Jun 16, 2026, 02:44 PM UTC

   Our coverage and analysis queues are erroring and delayed. We've narrowed the root cause down to our database vendor which ran a backwards incompatible upgrade.

2. monitoring Jun 16, 2026, 06:00 PM UTC

   We have remediated the source of the incident and are monitoring.

3. resolved Jun 16, 2026, 07:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 27, 2026, 05:51 PM UTC· resolved Apr 28, 2026, 01:14 AM UTC

1. identified Apr 27, 2026, 05:51 PM UTC

   We are currently investigating delays in Code Coverage processing. Coverage uploads that require mapping commits to pull requests may be delayed due to degraded performance in GitHub’s Pull Requests and Search APIs. This can impact the timely delivery of coverage statuses and comments on pull requests. Analysis processing is not affected. The issue is limited to coverage ingestion and reporting. We are actively monitoring the situation and will provide updates as GitHub recovers. For more information, see GitHub’s status page: https://www.githubstatus.com/: https://www.githubstatus.com/?utm_source=chatgpt.com

2. resolved Apr 27, 2026, 10:36 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 21, 2026, 09:19 PM UTC· resolved Apr 22, 2026, 02:48 PM UTC

1. investigating Apr 21, 2026, 09:19 PM UTC

   Commit statuses on PRs are currently delayed. We are actively investigating and will update this page when resolved.

2. monitoring Apr 21, 2026, 09:34 PM UTC

   We've scaled up additional capacity and the backlog of analysis results being pushed to GitHub has now cleared. We're continuing to monitor while we investigate the underlying cause of the earlier delays.

3. resolved Apr 21, 2026, 09:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 02, 2026, 10:22 PM UTC· resolved Apr 15, 2026, 05:19 PM UTC

1. identified Apr 02, 2026, 10:22 PM UTC

   A recent update has caused format builds to fail during Qlty Cloud analysis. We've identified the issue and are actively working on a fix. In the meantime, as a workaround, you can format your code locally (using `qlty fmt`) and push your changes as usual.

2. monitoring Apr 03, 2026, 12:34 AM UTC

   We have remediated the source of the incident and are monitoring.

3. resolved Apr 15, 2026, 05:19 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 30, 2026, 02:05 PM UTC· resolved Mar 30, 2026, 02:26 PM UTC

1. investigating Mar 30, 2026, 02:05 PM UTC

   We are investigating a queue backup that is impacting our ability to ingest and report on code coverage.

2. monitoring Mar 30, 2026, 02:26 PM UTC

   The coverage loader queue backlog has been cleared and processing has been restored. We drained the queue to restore full functionality, so coverage uploads made during the outage window will need to be re-triggered. We are monitoring to confirm stability. A full incident report will follow.

3. resolved Mar 31, 2026, 10:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 20, 2026, 04:59 PM UTC· resolved Apr 22, 2026, 02:46 PM UTC

1. monitoring Mar 20, 2026, 04:59 PM UTC

   Security Status Report: Trivy v0.69.4 Compromise Severity: Critical (upstream supply chain compromise) Status: Monitoring What Happened On March 19, 2026, a threat actor used a previously stolen service account credential to push a backdoored release of Trivy: https://github.com/aquasecurity/trivy (v0.69.4). The poisoned release distributed malicious binaries through GitHub Releases, Homebrew, AWS ECR, and rewrote 75 of 76 version tags on the trivy-action GitHub Action. The malware payload functions as an infostealer — targeting CI runner memory, SSH keys, and cloud credentials (AWS, GCP, Azure, and Kubernetes service account tokens). This is the second significant Trivy supply chain compromise in less than a month, and represents a meaningful escalation in both sophistication and reach from the first. Full technical details have been published by Boost Security Labs. The poisoned release (v0.69.4) has since been removed. The known-safe version is v0.69.3. Impact on Qlty Customers Qlty supports Trivy as an optional plugin. We have assessed our exposure across two customer segments: Qlty Cloud Customers — Not Impacted We queried all recorded Trivy plugin invocations across Qlty Cloud and found zero executions of v0.69.4. Qlty Cloud customers were not impacted by this compromise. CLI-Only Customers — Likely Not Impacted Qlty does not automatically pull new plugin releases. Customers who have enabled the Trivy plugin without customizing its version are running v0.69.2 — which predates the compromised release and is unaffected. However, Qlty's plugin system allows version overrides in your .qlty/qlty.toml file. If you manually specified v0.69.4 as your Trivy version, Qlty would have downloaded and executed the compromised binary. If you are a CLI-only user and are unsure whether you overrode this version, please check your .qlty/qlty.toml now. What We're Doing This is the second critical Trivy supply chain compromise in under a month. After the first incident, we removed Trivy from Qlty's recommended plugins during onboarding. Given this repeated pattern, and given that a supply chain compromise of this nature can be the leading edge of a broader attack campaign, we believe the responsible path forward is to recommend disabling the plugin. As with any static analysis tool, users are ultimately responsible for choosing which tools to run in their environment — Qlty operates under a shared responsibility model, and we want to give you the information you need to make that call. Recommended Action To disable the Trivy plugin, remove the Trivy entry from the [plugin] section of your .qlty/qlty.toml file: # Remove or comment out the following block: [[plugin]] name = "trivy" We will communicate when we are confident it is safe to re-enable Trivy. Questions If you have questions about your specific environment or believe you may have been impacted, please reach out to us at [email protected]: mailto:[email protected].

2. resolved Apr 22, 2026, 02:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 02, 2026, 03:30 PM UTC· resolved Mar 03, 2026, 02:52 PM UTC

1. identified Mar 02, 2026, 03:31 PM UTC

   Trivy experienced a security incident on 2026-03-01, which resulted in GitHub releases between v0.27.0 and v0.69.1 being deleted, causing build failures for any project using an affected version of the Trivy plugin. Workarounds: • Pin Trivy to v0.69.2 in your .qlty/qlty.toml • Or temporarily disable the Trivy plugin until the situation is resolved For more details on the upstream incident, see the Trivy security incident report: https://github.com/aquasecurity/trivy/discussions/10265.

2. monitoring Mar 02, 2026, 05:16 PM UTC

   CLI v0.615.0 has been released with the following fixes: • Trivy is no longer enabled by default when generating a new qlty.toml • Unpinned Trivy usage will now automatically use v0.69.2, the latest available release If you have Trivy pinned to a specific version in your .qlty/qlty.toml, you will need to either update it to 0.69.2 or disable the plugin until the upstream situation is resolved. For more details on the upstream incident, see the Trivy security incident report: https://github.com/aquasecurity/trivy/discussions/10265

3. resolved Mar 03, 2026, 02:52 PM UTC

   We've released a fix for this issue. Here's what you need to know: • No action needed if you don't have Trivy pinned to a specific version — Qlty will now automatically use the latest available release today (0.69.2). • If you have Trivy pinned in your qlty.toml, you can remove the pinned version or update it to 0.69.2 to restore builds immediately. • If you'd prefer not to change your config, you can temporarily disable the Trivy plugin to unblock your builds. You can track Trivy's resolution progress here: https://github.com/aquasecurity/trivy/discussions/10265

Read the full incident report →

Started Feb 24, 2026, 01:24 PM UTC· resolved Feb 24, 2026, 04:10 PM UTC

1. investigating Feb 24, 2026, 01:24 PM UTC

   We are investigating unexpected behavior.

2. monitoring Feb 24, 2026, 02:09 PM UTC

   We have remediated the source of the incident and are monitoring.

3. resolved Feb 24, 2026, 04:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 18, 2026, 04:56 PM UTC· resolved Feb 18, 2026, 09:04 PM UTC

1. investigating Feb 18, 2026, 04:56 PM UTC

   We are investigating reports of delayed analysis builds.

2. monitoring Feb 18, 2026, 05:44 PM UTC

   We have identified an issue causing delays in analysis builds. Our team has implemented a fix, and our systems are currently in the recovery phase. For any pull requests which haven't yet received analysis results, you can manually trigger a new build by pushing a new commit. We are monitoring the backlog to ensure all services return to normal as quickly as possible.

3. resolved Feb 18, 2026, 09:04 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 17, 2026, 04:51 PM UTC· resolved Feb 17, 2026, 09:06 PM UTC

1. investigating Feb 17, 2026, 04:51 PM UTC

   We are investigating reports of delayed analysis builds.

2. monitoring Feb 17, 2026, 06:17 PM UTC

   We have remediated the source of the incident and are monitoring.

3. resolved Feb 17, 2026, 09:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 27, 2026, 10:37 PM UTC· resolved Jan 27, 2026, 11:04 PM UTC

1. investigating Jan 27, 2026, 10:37 PM UTC

   We are investigating delays in publishing code quality analysis results back to GitHub.

2. identified Jan 27, 2026, 11:04 PM UTC

   We have identified the source of this incident and applied a partial fix. Quality analysis results should now be published to GitHub as expected.

3. monitoring Jan 27, 2026, 11:41 PM UTC

   We have remediated the source of the incident and are monitoring.

4. resolved Jan 28, 2026, 07:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 15, 2026, 05:31 PM UTC· resolved Jan 15, 2026, 07:45 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jan 09, 2026, 07:50 PM UTC· resolved Jan 09, 2026, 08:23 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Dec 23, 2025, 03:32 PM UTC· resolved Dec 23, 2025, 05:57 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Nov 18, 2025, 09:19 PM UTC· resolved Nov 18, 2025, 10:09 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Nov 03, 2025, 04:15 PM UTC· resolved Nov 03, 2025, 06:09 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Oct 20, 2025, 01:08 PM UTC· resolved Oct 21, 2025, 04:23 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Oct 09, 2025, 02:31 PM UTC· resolved Oct 15, 2025, 04:48 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →