Skip to main content

Tools

Python test framework — discipline gate for "done means tests pass."

Detection Post-generation Fully autonomous Program Stable Activity band C Environment
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: C (raw score 5.17). Shown band: C .

Facets: documented_in=0evaluated_on=0integration_surfaces=1.5paper_citations=0recent_push=2stars=1.67

Source of truth: pipeline/maturity/score.py

Description

pytest is the de facto Python test framework: it discovers and runs unit, integration, and functional tests, reporting failures in human-readable form. It is the usual gate for the "done means tests pass" rule: failing assertions show logic errors that only appear when code runs. Its summary also counts skipped and xfailed tests, which a person can read for unfinished work or weakened tests; pytest cannot tell a legitimate skip from a manipulated test. Configuration lives in pyproject.toml [tool.pytest.ini_options], which the audit detector now recognises.

Why it's included

Occupies detection × post_generation × fully_autonomous for logic_error in the Python lane. For incomplete_execution and test_manipulation it only records evidence (skip and xfail counts); the 2026-11-20 tag audit moved both to records_failure_modes (research/fm_tag_audit/final_table.csv). Mature workhorse; missing it leaves the audit detector unable to recognise the single most common testing configuration in Python repos.

Security notes

Standard dev-tool security posture; pytest executes user-authored test code with full developer privileges, so test files should be reviewed with the same scrutiny as production code. Plugin ecosystem is large and worth pinning by version.

Addresses failure modes

These tags are the catalog's claim about what pytest is built to catch or prevent. A tag is not a test result. SAICA-Bench has not measured pytest, so none of these tags is tested here.

Records evidence of

pytest can record evidence of these failures (traces, or checks you configure yourself), but it does not detect or prevent them on its own. They are not counted as failure modes it addresses.

Techniques

test-discoveryfixture-injectionparametrized-testingassertion-introspection

View raw YAML on GitHub →