Skip to main content

About

SAICA-Bench measures what reviewers flag when they are shown the work of AI coding agents, on 9 failure modes: fabricated code, test manipulation, obsolescence, scope creep and more. It runs free programs (type checkers, linters), packaged AI tools and plain language models on labelled samples and reports what each flagged, with a 95% interval and the cost. It is built by Vasyl Rakivnenko (Stanford University).

Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. 6 of the 9 sets have bad samples that were planted on purpose and 2 are real agent output; the benchmark page says how each set was made and lists the threats to validity, and the findings report says what the results do and do not show.

The failure-mode taxonomy and the tool catalog come from our knowledge graph of supervision tools (graph). A failure-mode tag on a tool is the catalog's claim about what the tool is built for; a benchmark row is a measurement of one configuration on one set. A tool that was benchmarked but is not in the catalog is marked as such. Code and results are on GitHub; these pages show SAICA-Bench results v2.0.0, built 22 November 2026.

Data sources

The samples come from SWE-rebench-OpenHands-Trajectories (Nebius) (public since 2025-09), SusVibes (LeiLiLab) (public since 2025-12), Collu-Bench (lt-asset) (public since 2024-10), Delulu (Microsoft) (public since 2026-03), Wang et al. deprecated-API map (ICSE'25) (public since 2024-06), OpenSSF malicious-packages (public since 2023-03) and Patronus TRACE (public since 2026-01). Per failure mode:

Code is Apache-2.0 and the results data CC-BY-4.0; the source datasets keep their own licenses.

Contact

Questions, corrections or a tool to add: open an issue or email Vasyl Rakivnenko at [email protected]. You can also find him on LinkedIn.