Is Qlty Software down?

Status: Investigating We are responding to the recently disclosed eMudhra certificate authority compromise: https://emudhra.we-have-a-problem.com/. An attacker able to intercept network traffic could impersonate trusted services to software that still accepts the affected certificates. We have released initial mitigations for our hosted analysis infrastructure. We are continuing to review and remediate Qlty services, build environments, the CLI, and its bundled tooling, including installation and upgrade paths. Customers using Qlty CLI or self-hosted CI should apply available operating-system security updates. Some developer tools maintain separate certificate bundles, so an operating-system or browser update alone may not provide complete protection. Keep certificate verification enabled. We will publish confirmed fixed Qlty versions, tested upgrade instructions, and any required runtime or cache refresh steps as they become available. We will update this notice as remediation progresses.

Status: Resolved This incident has been resolved.

Status: Identified Since 10:35 AM ET (14:35 UTC) on October 8, 2026, analysis builds fail for projects that use the Semgrep plugin. The build stops while installing Semgrep, before any analysis runs. Projects that do not use Semgrep are not affected. Coverage uploads are not affected. A workaround is available now (below). We are sorry for the disruption. What is happening Semgrep 1.144.0, the version Qlty installs by default, depends on a package called mcp that imports a private function from pydantic. pydantic 2.14.0, published at 10:34 AM ET (14:34 UTC) today, removed that function. Semgrep does not limit which pydantic version it installs, so every fresh install of Semgrep now picks up pydantic 2.14.0 and fails to start. The build log shows: ImportError: cannot import name 'eval_type_backport' from 'pydantic._internal._typing_extra' Semgrep releases 1.137.0 through 1.145.0 share this dependency and fail the same way. Releases 1.146.0 and later install correctly. Who is affected - Qlty Cloud: every analysis build for a project with Semgrep enabled fails while installing tools. Affected pull requests do not get an analysis result. - qlty CLI in CI, or on any machine installing Semgrep for the first time: `qlty check` fails while installing Semgrep. - qlty CLI on a machine where Semgrep was installed before 10:34 AM ET today: not affected. The installed copy keeps working until the tool cache is cleared. Workaround, available now in Semgrep to 1.179.0 in `.qlty/qlty.toml`: [[plugin]] name = "semgrep" version = "1.179.0" If your `qlty.toml` already has a `[[plugin]]` entry for `semgrep`, add or change its `version` line. Commit and push. The next Qlty Cloud build and the next `qlty check` run install 1.179.0, which works with pydantic 2.14.0. If you had already pinned Semgrep to a version between 1.137.0 and 1.145.0, you need to make this change yourself. Our fix changes the default version only and does not override a version you set. Semgrep 1.179.0 is 35 releases newer than 1.144.0. Its rules have changed in that time, so you may see new findings or lose some old ones after the switch. We chose 1.179.0 over a smaller jump because every Semgrep release between 1.146.0 and 1.178.0 depends on packages with open security advisories. What we are doing We are changing the default Semgrep version in the qlty CLI from 1.144.0 to 1.179.0 (https://github.com/qltysh/qlty/pull/2910). We will ship it in the next CLI release and roll it out to Qlty Cloud. After that, projects without an explicit Semgrep version build again with no change on your side. Builds that failed during the incident do not rerun on their own. Push a new commit to get a fresh result. Affected components Code Analysis (Partial outage)

Status: Resolved This incident has been resolved. Affected components Code Analysis (Operational)

Status: Resolved On October 5, 2026, between 12:18 and 12:31 PM ET, Qlty stopped completing analysis results for 13 minutes. About 120 builds across roughly 60 workspaces never completed. On the affected pull requests, the Qlty check on GitHub showed "Qlty encountered a timeout error at 15mins" while the build page on Qlty showed the build as passed. Pushing a new commit to an affected pull request produces a fresh, correct result. We are sorry for the confusion this caused. Below is what happened, who was affected, why, and what we are changing. Timeline (US Eastern on October 5, 2026, UTC in parentheses) • 12:17 PM (16:17): A routine data cleanup in one workspace started a large maintenance operation on our main database. • 12:18 PM (16:18): Analysis results stopped completing. Builds kept running and finishing normally, but their results were not written. • 12:30 PM (16:30): Our queue-delay alarm fired. • 12:31 PM (16:31): The operation finished and results started completing again. • 12:33 to 12:50 PM (16:33 to 16:50): The Qlty check on affected pull requests changed from pending to the 15-minute timeout error. • 12:35 PM (16:35): Result processing was back to normal. • About 1:00 PM (17:00): A customer reported a pull request stuck on the timeout error. We advised pushing a new commit. • 4:00 PM (20:00): We confirmed the root cause. Who was affected During the 13-minute window, every analysis result was delayed by up to 12 minutes. Most builds recovered on their own: our queue retries each result three times, and 73 builds completed on a later attempt. 118 builds did not complete after three attempts and were dropped. They span 88 pull requests and 26 default-branch pushes, across about 80 projects in roughly 60 workspaces. For each of these builds: • The Qlty check on GitHub showed the timeout error instead of a pass or fail, and the check linked to our troubleshooting docs rather than to the build. • No pull request comment or review was posted. • The build page on Qlty showed the build as passed, with no indication that its results were missing. • For the default-branch pushes, that commit is missing from the project's quality history until the next push. A smaller group of builds that did complete on a retry recorded some issues more than once, which inflated their issue counts. This affects 42 builds. We are deciding whether to correct these or leave them, since a later push replaces them. If you had a pull request open during this window and the Qlty check still shows the timeout error, push a new commit and the check will run again. Root cause A customer ran a routine cleanup of older data in their project. That is a normal, supported action. Behind it, Qlty removes that project's data from one of our largest tables. Because of how that table is laid out, removing one project's data required reading a large share of the whole table to find the small fraction that matched. That read did not slow the database itself as much as it slowed ClickHouse Keeper, the coordination service that tracks what data each database replica has. Every request to Keeper became about six times slower for the 13 minutes the operation ran. The final step of processing an analysis result, which computes a build's metrics, asks Keeper to confirm the replica has the latest data before the query starts. Under normal conditions that check takes under a second. With Keeper slowed, and about 40 of these steps waiting at once, each check queued behind the others and the wait grew to several minutes. Our 30-second client timeout cut each attempt short, the queue retried three times over about six minutes, and builds whose three attempts all fell inside the 13-minute window were dropped. What we are doing • Making data removal a bounded operation. Removing a project's data will cost in proportion to the data being removed, not the size of the table it lives in, so one workspace's cleanup cannot slow the service for everyone. • Reducing the final processing step's dependence on Keeper. We are cutting the number of coordination checks each metrics query makes, and we are designing a version of that step that does not need them at all. • Alerting sooner. We are adding alerts on the database timeouts that preceded the queue delay, so we see an event like this within a couple of minutes instead of twelve. If you have questions about a specific pull request or build from this window, contact us through qlty.sh/contact/support: http://qlty.sh/contact/support and include the pull request or build link.

Status: Identified We have identified the source of this incident, GitHub's various APIs are experiencing degrading performance https://www.githubstatus.com/incidents/zkxwbgr0cnmx. Affected components Code Analysis (Degraded performance) Code Coverage (Degraded performance) GitHub.com (Degraded performance)

Status: Resolved This incident has been resolved. Affected components Code Analysis (Operational) Code Coverage (Operational) GitHub.com (Operational)

Status: Monitoring We are currently investigating intermittent build failures caused by network issues when downloading plugins from GitHub during analysis runs. Our team is actively identifying the root cause and working on a resolution. We will provide an update as soon as more details or a fix are available. Affected components Code Analysis (Partial outage)

Status: Identified Since approximately 9:30 AM PT, a portion of analysis builds have been failing during the tool-installation step, with errors like "Error installing tool …" in the build log. We've traced this to an upstream disruption affecting downloads of release assets from GitHub — the archives our builds fetch to install linters and language runtimes. The disruption is intermittent: during failure bursts roughly 1 in 8 builds fails, and between bursts builds complete normally. We have escalated it to GitHub. What you can do: re-running a failed build will usually succeed, though a retry may still hit a failure window while the disruption continues. No action is needed for builds that completed successfully. Coverage uploads and processing, qlty.sh: http://qlty.sh, and the API are unaffected. We're monitoring failure rates continuously and will post an update when the disruption clears. Affected components Code Analysis (Partial outage)

Status: Identified We have identified the source of this incident and are working on remediation. Affected components Code Analysis (Partial outage) GitHub.com (Partial outage)

Status: Resolved GitHub acknowledged that this was an incident: https://www.githubstatus.com/incidents/lsvy8xsf0gxv and resolved it. Affected components Code Analysis (Operational) GitHub.com (Operational)