Skip to main content

Deprecated API usage (obsolescence) in AI-generated code

Obsolescence is when an AI coding agent calls a real library API or package that has been deprecated, removed or materially changed, usually because the model’s training data predates the current release. The model saw an older version of the library and confidently writes the older form. It differs from code hallucination, where the API never existed, and from dependency blindness, where a current API exists and is ignored.

Old calls break when the library moves on. Wang et al. gave seven code LLMs completion prompts built on 145 deprecated APIs from eight Python libraries: of the completions that used either the old API or its replacement, 25% to 38% used the deprecated one.

At a glance: On SAICA-Bench: No single best reviewer: 3 rows cannot be told apart at 95% on the samples they share (gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash); the highest J is +89.7% (95% interval +82.9% to +95.1%). Results ↓

Also called
  • stale training knowledge
  • training cutoff drift
  • version drift
  • deprecated API usage
  • library evolution lag
  • training data staleness
  • deprecated API
  • outdated reference
  • stale knowledge
  • API drift
  • knowledge cutoff drift
  • library evolution
  • deprecated method

How well can deprecated API usage be caught?

J 80% or more on our set The code uses a deprecated or removed library API. No single best reviewer: 3 rows cannot be told apart at 95% on the samples they share (gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash); the highest J is +89.7% (95% interval +82.9% to +95.1%).

Best reviewer

any ranked row; a tie is shown as a top group

top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash

Highest J +89.7% (95% interval +82.9% to +95.1%)

Best catalog tool

a tool of the catalog running its own check

pyright (latest libraries)

J +56.8% (95% interval +42.0% to +71.2%)

Shortcut baseline

a rule that understands nothing about the failure

The code uses an API listed in the 2024 deprecation map

J +97.4%. Read the reviewers against this, not against 0.

What was tested

Bad samples
Failure planted on purpose. an automatic code edit that swaps the deprecated call into solutions written by five models; four LLM-SDK pairs written by hand. Label: construction; the swapped solutions were run against the latest libraries. Clean samples: the untouched solutions of the same five models.
One sample
one solution; solutions for the same deprecated API are grouped (intervals resample API pairs).
Samples scored
155 solutions calling a deprecated API and 155 clean (50% of the set is bad).
Measure
J 80% or more on our set.
Shortcut baseline
A rule with no understanding of the failure scores J +97.4% here (The code uses an API listed in the 2024 deprecation map). Read reviewer scores against it, not against 0.

Threats to validity that apply to this set →

All results on this set

Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

Obsolescence: ranked reviewers, best first
# Reviewer 95% interval bad samples flagged clean samples flagged No verdictanswers left out False alarmsper 100 clean changes Precisionif 1% of changes are badPrecisionif 5% of changes are badPrecisionif 20% of changes are bad per 1,000 reviews
1 gpt-6-luna, asked directly
Model asked directly · in the top group
Run card
Model
gpt-6-luna
Provider
Azure AI Foundry
What it was shown
task text; code
Counts as flagged when
flagged when the answer carries "uses_deprecated_api": true
Output token limit
16384
First run
2026-11-20
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.17 per 1,000 reviews (list); $0.05 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+89.7%+82.9% to +95.1% 97.4%93.6% to 99.0% 7.7%4.5% to 13.0% 0 7.7 11%40%76% $0.17list
2 mimo-v2.6-flash, asked directly
Model asked directly · in the top group
Run card
Model
mimo-v2.6-flash
Provider
Hyperbolic
What it was shown
task text; code
Counts as flagged when
flagged when the answer carries "uses_deprecated_api": true
Temperature
0
Output token limit
32768
First run
2026-11-20
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.25 per 1,000 reviews (api); $0.08 spent on this run. Charge reported by the API with each call (Hyperbolic)
+88.4%+80.5% to +94.7% 91.6%86.2% to 95.0% 3.2%1.4% to 7.3% 0 3.2 22%60%88% $0.25api
3 deepseek-v4-flash, asked directly
Model asked directly · in the top group
Run card
Model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
task text; code
Counts as flagged when
flagged when the answer carries "uses_deprecated_api": true
Temperature
0
Output token limit
32768
First run
2026-11-20
Runs
310; failed 2 (0.6%): truncated 2
Samples
155 bad, 155 clean
Answers with no verdict
2 (1 bad, 1 clean): truncated 2
Cost
$0.52 per 1,000 reviews (api); $0.16 spent on this run. Charge reported by the API with each call (Hyperbolic)
+88.3%+81.9% to +93.9% 97.4%93.5% to 99.0% 9.1%5.5% to 14.7% 2 of 310 9.1 10%36%73% $0.52api
4 pyright against the latest libraries (deprecated, missing attribute or import)
Catalog tool · Free program · own check
Run card
Version
pyright 1.1.414
What it was shown
code; latest libraries installed
Counts as flagged when
Flagged when pyright reports reportDeprecated, reportAttributeAccessIssue or reportMissingImports (any severity) with the latest libraries installed.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • reportDeprecated raised to error; counted: reportDeprecated, reportAttributeAccessIssue, reportMissingImports
  • run with the latest release of each library installed
+56.8%+42.0% to +71.2% 60.6%52.8% to 68.0% 3.9%1.8% to 8.2% 0 3.9 14%45%80% $0free
5 grok-4.1-fast (no reasoning), asked directly
Model asked directly
Run card
Model
grok-4-1-fast-non-reasoning
Provider
Azure AI Foundry
What it was shown
task text; code
Counts as flagged when
flagged when the answer carries "uses_deprecated_api": true
Temperature
0
Output token limit
8192
First run
2026-11-20
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.12 per 1,000 reviews (list); $0.04 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+52.9%+40.9% to +64.3% 90.3%84.6% to 94.0% 37.4%30.2% to 45.3% 0 37.4 2%11%38% $0.12list
6 mypy against the latest libraries (deprecated, missing attribute or import)
Catalog tool · Free program · own check
Run card
Version
mypy 2.4.1
What it was shown
code; latest libraries installed
Counts as flagged when
Flagged when mypy reports an error coded deprecated, attr-defined or import-not-found with the latest libraries installed.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • error code 'deprecated' switched on; counted: deprecated, attr-defined, import-not-found
  • run with the latest release of each library installed
+14.2%+6.6% to +23.4% 15.5%10.6% to 22.0% 1.3%0.4% to 4.6% 0 1.3 11%39%75% $0free
7 Phi-4-mini, asked directly
Model asked directly
Run card
Model
Phi-4-mini-instruct
Provider
Azure AI Foundry
What it was shown
task text; code
Counts as flagged when
flagged when the answer carries "uses_deprecated_api": true
Temperature
0
Output token limit
1024
First run
2026-11-20
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.03 per 1,000 reviews (list); $0.01 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+14.2%+5.6% to +24.0% 14.8%10.1% to 21.3% 0.6%0.1% to 3.6% 0 0.6 19%55%85% $0.03list
8 ruff NumPy rules (NPY)
Catalog tool · Free program · own check
Run card
Version
ruff 0.17.6
What it was shown
code
Counts as flagged when
Flagged on any NPY finding.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • NPY rules only; any finding counts
+7.7%0.0% to +17.0% 7.7%4.5% to 13.0% 0.0%0.0% to 2.4% 0 0 100%100%100% $0free
9 ruff pyupgrade rules (UP)
Catalog tool · Free program · own check
Run card
Version
ruff 0.17.6
What it was shown
code
Counts as flagged when
Flagged on any UP finding.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • UP rules only; any finding counts
+1.3%−4.4% to +7.2% 7.1%4.0% to 12.3% 5.8%3.1% to 10.7% 0 5.8 1%6%23% $0free

Shown for reference, not ranked

These rows are never counted as best. A shortcut baseline shows what a rule with no understanding of the failure scores on this set; read the reviewers' scores against it.

Obsolescence: rows shown for reference and not ranked
Row and why it is not ranked J95% interval TPR FPR No verdict Costper 1,000 reviews
Construction check: the code uses an API listed in the 2024 deprecation map
Shortcut baseline · construction check
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
The builder's own matcher against the map the positives were made from (108 usable pairs of Wang et al., ICSE 2025). It catches every swapped-in call by construction and has no entry for the four SDK retirements of 2025. A lookup knows what is in its list, nothing newer.
+97.4%+94.4% to +99.4% 97.4% 0.0% 0 $0
Leak check: two `from M import` lines for the same module in a row
Shortcut baseline · leak check
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
A format trace of the swap tool: when it renames one imported name it writes a second import line instead of editing the first. It reads no API name.
+7.7%+2.0% to +15.2% 7.7% 0.0% 0 $0
How to read these numbers
  • J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
  • An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
  • 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
  • Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
  • A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
  • Precision is the share of flagged changes that are really bad, if the stated share of all changes were bad. It is computed from the row's TPR and FPR; the set's own share of bad samples is 50%.

The same answers, scored another way

Parts of the set, or the same set under another labelling. They show how far the headline J depends on how the set was built.

Map renames 2024 · 151 bad, 151 clean

Solutions with a modern call swapped back to its deprecated twin, against untouched solutions. Every API change here is in a map published in 2024 and is years old: older than what any reviewed model has read.

RowJ95% intervalTPRFPR
gpt-6-luna+91.4%+84.8% to +96.5%99.3%8.0%
mimo-v2.6-flash+90.1%+82.3% to +96.2%93.4%3.3%
deepseek-v4-flash+90.0%+83.9% to +95.3%99.3%9.3%
pyright (latest libraries)+57.6%+42.4% to +72.4%61.6%4.0%
grok-4.1-fast+54.3%+42.1% to +65.9%90.1%35.8%
Phi-4-mini+14.6%+5.8% to +24.5%15.2%0.7%
mypy (latest libraries)+14.6%+6.8% to +24.1%15.9%1.3%
ruff NPY rules+8.0%0.0% to +17.6%8.0%0.0%
ruff UP rules+1.3%−4.5% to +7.5%7.3%6.0%
Sdk retirements 2025 · 4 bad, 4 clean

The four hand-written LLM-SDK pairs: a model id retired on 2025-10-28, a parameter removed from an SDK, a legacy SDK whose support ended on 2025-11-30. These are the only samples that may be newer than a reviewer's training data (cut-off dates are not published for all five models). n = 4 + 4; two of the four clean twins are the same code.

RowBad samples flaggedClean samples flagged
gpt-6-luna1 of 40 of 4
deepseek-v4-flash1 of 40 of 4
mimo-v2.6-flash1 of 40 of 4
pyright (latest libraries)1 of 40 of 4
grok-4.1-fast4 of 44 of 4
Phi-4-mini0 of 40 of 4
ruff NPY rules0 of 40 of 4
ruff UP rules0 of 40 of 4
mypy (latest libraries)0 of 40 of 4
The code uses an API listed in the 2024 deprecation map0 of 40 of 4

Counts, not rates: a class of this part has fewer than 20 samples.

Clean side checked · 155 bad, 150 clean

The whole set without the 5 'clean' samples that use another deprecated or removed API (table clean_samples_checked).

RowJ95% intervalTPRFPR
gpt-6-luna+92.1%+86.0% to +96.8%97.4%5.3%
mimo-v2.6-flash+91.0%+83.8% to +96.6%91.6%0.7%
deepseek-v4-flash+90.7%+84.9% to +95.7%97.4%6.7%
pyright (latest libraries)+58.0%+43.1% to +72.4%60.7%2.7%
grok-4.1-fast+55.0%+43.0% to +66.3%90.3%35.3%
Phi-4-mini+14.2%+5.6% to +24.0%14.8%0.7%
mypy (latest libraries)+14.1%+6.6% to +23.2%15.5%1.3%
ruff NPY rules+7.7%0.0% to +17.2%7.7%0.0%
ruff UP rules+1.1%−4.7% to +7.1%7.1%6.0%
Without tensorflow · 134 bad, 135 clean

Everything but the TensorFlow samples. pyright and mypy have no usable type information for TensorFlow, so this is the part of the set they can check.

RowJ95% intervalTPRFPR
mimo-v2.6-flash+92.5%+85.9% to +97.4%94.0%1.5%
gpt-6-luna+90.3%+83.4% to +95.7%97.0%6.7%
deepseek-v4-flash+88.8%+82.2% to +94.5%97.0%8.2%
pyright (latest libraries)+65.7%+50.0% to +80.1%70.2%4.4%
grok-4.1-fast+55.5%+42.2% to +68.0%88.8%33.3%
mypy (latest libraries)+16.4%+7.7% to +27.0%17.9%1.5%
Phi-4-mini+16.4%+6.6% to +27.3%17.2%0.7%
ruff NPY rules+9.0%0.0% to +19.6%9.0%0.0%
ruff UP rules+0.8%−5.7% to +7.5%7.5%6.7%
More tables for this set in results.json
  • positives_by_current_status: What the deprecated API of each positive does in the installed latest release (check_map.py calls each one). A name can still exist and raise 'is now removed' when called.
  • clean_samples_checked: Every clean sample that two or more of the three strong models flag, or that a type checker flags for a library API, was checked against the installed release (check_clean_side.py), plus every TensorFlow name. 'not_confirmed' means the reviewers' objection does not hold in that release.
  • sdk_pairs: The four hand-written SDK pairs: deprecated samples caught, and clean twins flagged. ob-h0006 and ob-h0008 (two clean twins) are the same code.
  • type_checkers_by_library
  • type_checkers_after_tensorflow_install
  • own_code_vs_other_models_code: Each reviewed model also wrote part of the solutions. J on the samples a model wrote itself and on the samples the other four wrote (hand-written pairs left out). Each difference alone is within its interval.

They are under modes[].tables of results.json.

How it was tested

  • 151 solutions with the modern API swapped back to its deprecated twin vs 151 untouched solutions, all written by the same 5 models (gpt-6-luna, grok-4.1-fast, Phi-4-mini, deepseek-v4-flash, mimo) for the same tasks, across numpy, pandas, scipy, scikit-learn, seaborn, PyTorch, TensorFlow and transformers.
  • The swap is an automatic code edit (imports, aliases, dotted paths, method and class names), kept only if the result uses the deprecated API and nothing else from the map. Checked against the latest release of each library: 114 of the swapped solutions use a name that is gone, 24 call a function that still exists but raises when called, and 13 still run with a deprecation notice.
  • Plus 4 hand-written pairs for LLM SDKs (155 + 155 samples in all). Anthropic: a call that names Claude 3.5 Sonnet (retired Oct 28, 2025) and passes temperature (removed from messages.create in SDK 1.x) through a settings object, and a modern streaming call that still names the retired model. Google: a streaming tutor chatbot on the legacy google-generativeai SDK (support ended Nov 30, 2025; imported as `from google import generativeai as genai`) with gemini-pro, and the same chatbot on the current google-genai SDK but still naming gemini-pro. Clean twins use claude-sonnet-5-5 / gemini-3.8-flash on the current SDKs.
  • Reviewers see the task and the code and are told to assume the latest library versions. Same prompt for every model; temperature 0 (gpt-6-luna: its default). Run 2026-11-20.

Caveats

  • Synthetic: the deprecated calls were swapped in; models writing freely today rarely use these APIs (49 of 956 solutions).
  • The KG's type checkers were run with the latest version of every library installed (numpy 2.5, pandas 3.1, scipy 1.18, scikit-learn 1.9, PyTorch 2.15, TensorFlow 2.21, transformers 5.23, current Anthropic and Google SDKs): they catch removed APIs (missing attribute or import) and APIs marked @deprecated; mypy finds far fewer because many libraries ship no type information for it. They cannot see a retired model name. The other KG tools tagged obsolescence work before code is written or bump dependency versions. ruff's two rule sets for old APIs cover only NumPy 2 (NPY) and Python-syntax upgrades (UP).
  • Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
  • Answers with no verdict are left out of the rates; they are not counted as 'not flagged'. Unreadable: deepseek-v4-flash 2 of 310 (1 bad, 1 clean; mostly answers cut by the token budget).
  • 151 of the 155 positives are renames from a deprecation map published in 2024; on those the three strong models catch 99% (gpt-6-luna), 99% (deepseek-v4-flash), 93% (mimo-v2.6-flash). The case that defines this failure, an API retired after the reviewer's knowledge ends, has 4 hand-written pairs: gpt-6-luna caught 1 of 4, deepseek-v4-flash caught 1 of 4, mimo-v2.6-flash caught 1 of 4; grok-4.1-fast flagged all 4 and 4 of the 4 clean twins. Four pairs cannot give a rate; the headline J is about old renames.
  • A lookup in the 2024 map scores J +97%: it catches all 151 swapped-in calls and none of the 4 SDK retirements. Most of this set measures whether a reviewer knows that map.
  • The clean side is not fully clean: 5 of the 155 'clean' samples use another deprecated or removed API (3 use TensorFlow 1 names that TensorFlow 2 removed, 2 a deprecated seaborn argument or property). They are 4 of gpt-6-luna's 12 false alarms, 4 of deepseek-v4-flash's 14 false alarms, 4 of mimo-v2.6-flash's 5 false alarms. Without them J is +92% (gpt-6-luna), +91% (deepseek-v4-flash), +91% (mimo-v2.6-flash). 3 more flagged clean samples were checked and are clean: the API the reviewers name runs without a warning in the installed release.
  • pyright and mypy cannot check TensorFlow: they flag 0 of the 21 deprecated and 0 of the 20 clean TensorFlow samples, before and after TensorFlow 2.21.0 was installed in their environment (pyright reads partial bundled stubs, mypy has no type information). On the other 269 samples pyright scores J +66% and mypy +16%.
  • What the positives' APIs do today: 114 names are gone from the latest release, 24 are PyTorch functions that still exist as names but raise 'is now removed' when called, 13 still run with a deprecation notice, and 4 are the SDK pairs.
  • The five reviewed models also wrote the solutions. 4 of 5 score higher on their own code than on the others' (table); a reviewer that wrote no sample has not been added.
  • 12 swapped solutions and 0 untouched ones carry a format trace of the swap tool (two import lines for one module); the rule that reads only that scores J +8%. The deprecated side is synthetic: each clean solution appears on one side only, so the classes are not pairs.

Obsolescence on the leaderboard → How this set was built → Threats to validity → Build a supervision kit →

How to detect it

Signals that point to obsolescence, from the SAICA knowledge graph:

  • Imported API is marked deprecated in the target library's current release notes.
  • Package pinning is older than the latest minor release by N versions (threshold configurable).
  • Language-server or linter emits a deprecation warning.
  • Behavior differs between training-era and current versions of the library.

Studies and reported patterns

Records in the catalog that are not single dated incidents: findings of a study, recurring behaviour that was reported, or scenarios a source describes as hypothetical.

Tools in the catalog that address it

8 tools are tagged with obsolescence in the catalog, most-starred first. A tag is the catalog's claim about what a tool is for; it is not a test result. Where SAICA-Bench ran the tool on this mode, the measured J is shown: that row is the evidence.

  • Context7 Up-to-date, version-pinned library documentation for coding agents — delivered as an MCP server.
  • Ruff measured J +7.7% Extremely fast Python linter and formatter, written in Rust.
  • Renovate Highly configurable multi-platform dependency update bot.
  • mypy measured J +14.2% Static type checker for Python. Catches mismatched types before runtime.
  • Pyright measured J +56.8% Microsoft's fast static type checker for Python.
  • Context Hub Up-to-date, curated API docs for coding agents.
  • Dependabot Automated dependency-update PRs that flag outdated and vulnerable packages.
  • Socket Dependency supply-chain scanner that inspects packages before install.

In other taxonomies

How obsolescence maps to other failure taxonomies. “Exact” means the same failure, “partial” an overlap, “related” a neighbouring category.

  • Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    • partial SHAH-RC-02 Dependency Drift — Dependencies "introduce breaking changes, deprecate interfaces, or alter platform support", so existing code calls an interface that has changed. Same observable as obsolescence. Different cause: here the library moved after the code was written; in obsolescence the agent writes the old form from stale training data.

    Related: SHAH-CAT-10 Dependency & Environment Management; SHAH-RC-04 Model Interface Instability; SHAH-RC-07 External API Instability

  • Code hallucination (fabrication) — Code that calls functions, methods or packages that do not exist.
  • Reinventing existing code (dependency blindness) — Hand-written code for what the standard library, a dependency or the codebase already provides.

All 11 AI coding agent failure modes → · Glossary

Research

Questions and answers

What is deprecated API usage (obsolescence) in AI coding agents?

Obsolescence is when an AI coding agent calls a real library API or package that has been deprecated, removed or materially changed, usually because the model’s training data predates the current release. The model saw an older version of the library and confidently writes the older form. It differs from code hallucination, where the API never existed, and from dependency blindness, where a current API exists and is ignored.

How do you detect deprecated API usage?

Signals that point to it: Imported API is marked deprecated in the target library's current release notes. Package pinning is older than the latest minor release by N versions (threshold configurable). Language-server or linter emits a deprecation warning. Behavior differs between training-era and current versions of the library. On SAICA-Bench: No single best reviewer: 3 rows cannot be told apart at 95% on the samples they share (gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash); the highest J is +89.7% (95% interval +82.9% to +95.1%).

Which reviewers catch deprecated API usage on SAICA-Bench, and how well?

No single best reviewer: 3 rows cannot be told apart at 95% on the samples they share (gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash); the highest J is +89.7% (95% interval +82.9% to +95.1%). The best catalog tool that ran its own check is pyright (latest libraries), at J +56.8%. A rule with no understanding of the failure scores J +97.4% on the same set (The code uses an API listed in the 2024 deprecation map), so read the scores against that, not against 0. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

How was detection of deprecated API usage measured?

On 310 scored samples: 155 solutions calling a deprecated API and 155 clean ones. Where the bad samples come from: failure planted on purpose (an automatic code edit that swaps the deprecated call into solutions written by five models; four LLM-SDK pairs written by hand). Label: construction; the swapped solutions were run against the latest libraries. Data: Wang et al. (ICSE'25) deprecated-API map (65 pairs, 8 libraries) + solutions written by 5 models, deprecated call swapped in. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Benchmark results from 2026-11-22. The definition, detection signals and tags come from the SAICA knowledge graph.

View raw YAML on GitHub →