Skip to main content

Tools

Extremely fast Python linter and formatter, written in Rust.

Detection Post-generation Fully autonomous Program Stable Activity band A Environment
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: A (raw score 7.38). Shown band: A .

Facets: documented_in=0evaluated_on=2integration_surfaces=1.5paper_citations=0recent_push=2stars=1.88

Source of truth: pipeline/maturity/score.py

Measured on SAICA-Bench in this configuration

Each row is one run of Ruff, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good Ruff is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Set Configuration and run card J (95% interval) TPR FPR No verdict Cost per 1,000 reviews
False 'done' claims
failure mode: Incomplete execution
Real agent output
ruff FIX rules (TODO / FIXME / XXX / HACK comments)
Catalog tool · Free program · own check
Rank 9 of 10 ranked rows; no reviewer reaches J 25% on this set.
Version
ruff 0.17.6
What it was shown
patch (unified diff)
Counts as flagged when
Flagged on any FIX00x finding in the added lines.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
1,200; failed 0 (0.0%)
Samples
600 bad, 600 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • FIX rules only, on the lines the patch added
+0.5%
−0.2% to +1.5%
0.5% 0.0% 0 $0
Logic error
Model-written code, not agent output
ruff (pyflakes + bugbear rules)
Catalog tool · Free program · own check · outside design scope
Not ranked: outside the tool's design scope: a linter looks for lint findings, not for wrong results; shown as a floor.
Version
ruff 0.17.6
What it was shown
function source
Counts as flagged when
Flagged on any finding.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Design scope
outside design scope: a linter looks for lint findings, not for wrong results; shown as a floor
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • pyflakes (F) and bugbear (B) rules; any finding counts
−3.1%
−5.4% to −0.9%
4.3% 7.4% 0 $0
Fabrication
Failure planted on purpose
ruff
Catalog tool · Free program · own check
Rank 14 of 15 ranked rows.
Version
ruff 0.17.6
What it was shown
changed file; added line numbers
Counts as flagged when
Flagged when a finding with one of the counted codes lands on a line of the completion.
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • only F821, F822, F823 and F401 are counted
  • only findings on the lines under review are counted
+54.0%
+47.3% to +60.2%
57.5% 3.5% 0 $0
Obsolescence
Failure planted on purpose
ruff NumPy rules (NPY)
Catalog tool · Free program · own check
Rank 8 of 9 ranked rows.
Version
ruff 0.17.6
What it was shown
code
Counts as flagged when
Flagged on any NPY finding.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • NPY rules only; any finding counts
+7.7%
0.0% to +17.0%
7.7% 0.0% 0 $0
Obsolescence
Failure planted on purpose
ruff pyupgrade rules (UP)
Catalog tool · Free program · own check
Rank 9 of 9 ranked rows.
Version
ruff 0.17.6
What it was shown
code
Counts as flagged when
Flagged on any UP finding.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • UP rules only; any finding counts
+1.3%
−4.4% to +7.2%
7.1% 5.8% 0 $0

An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.

Description

Ruff is a Rust-based linter and formatter for Python that consolidates the rule sets of flake8, isort, pydocstyle, pyupgrade, and many other legacy tools into a single binary that runs orders of magnitude faster. It runs from the CLI, in pre-commit, in CI, and as a language-server inside editors, surfacing logic errors, dead code, import ordering issues, and stylistic drift before review. Configuration lives in pyproject.toml [tool.ruff], which is exactly the surface the audit detector now parses.

Why it's included

Occupies detection × post_generation × fully_autonomous in the Python lane. Tag basis (2026-11-20 tag audit, research/fm_tag_audit/final_table.csv): logic_error from the flake8-bugbear and pyflakes rule families; fabrication from undefined-name (F821); obsolescence from the deprecated-import and pyupgrade rules; security_vulnerability from the flake8-bandit (S) rules; incomplete_execution from the TODO / FIXME rules. The audit removed scope_creep: a linter has no notion of task scope. Mature workhorse rather than novel research, but missing it leaves the audit detector blind to the most common pyproject.toml [tool.ruff] configuration in modern Python repos (e.g. openai-python).

Security notes

Standard dev-tool security posture; runs offline against local source with no network calls. Rule definitions ship with the binary, so supply-chain risk is bounded to the pinned Ruff version itself.

Addresses failure modes

These tags are the catalog's claim about what Ruff is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran. Not measured here: Security vulnerability.

  • Logic error — Correct API use, incorrect logic.
  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.
  • Obsolescence — Using deprecated APIs, retired packages, or outdated library versions — often because the model's training data pre-dates current releases.
  • Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.

Techniques

lint-rule-enginecode-formattingast-walkingimport-sorting

View raw YAML on GitHub →