Skip to main content

Tools

Uncertainty Quantification for Language Models.

Detection Post-generation Graduated HITL AI-based Experimental Activity band D ModelEnvironment
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: D (raw score 5.73). Shown band: D .

Facets: documented_in=0evaluated_on=2integration_surfaces=0.5paper_citations=0recent_push=2stars=1.23

capped from B to D by maturity_status='experimental'

Source of truth: pipeline/maturity/score.py

Measured on SAICA-Bench in this configuration

Each row is one run of uqlm, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good uqlm is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Set Configuration and run card J (95% interval) TPR FPR No verdict Cost per 1,000 reviews
Logic error
Model-written code, not agent output
uqlm CodeGenUQ: 5 fresh solutions, functional-equivalence rate < 0.5 (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 5 of 7 ranked rows.
Version
uqlm 0.6.8
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
task description; function source
Counts as flagged when
Flagged when fewer than half of 5 fresh solutions are judged functionally equivalent to the reviewed function.
Temperature
1
Output token limit
8192
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
600; failed 23 (3.8%): nan score 23
Samples
a subset: 300 bad and 300 clean of the set's 1,536 and 904
Answers with no verdict
23 (1 bad, 22 clean): nan score 23; under 95% of one class was readable
Cost
~$3.81 per 1,000 reviews (api); $2.28 spent on this run. Estimated: run on the first 300 + 300 samples only (our cost $2.28), scaled to all 2440; rates are on those 600. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • CodeGenUQ's functional-equivalence scorer: 5 fresh solutions written at temperature 1.0 are judged against the reviewed function
  • flagged when fewer than half are judged equivalent (cut-off 0.5 chosen by us)
+52.5%
+44.4% to +60.1%
96.0% 43.5% 23
incomplete
~$3.81

An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.

Description

UQLM (by CVS Health) is a Python package for uncertainty quantification of LLM outputs. Implements black-box, white-box, and semantic-similarity-based uncertainty estimators so downstream code can abstain, escalate to human review, or request clarification when confidence is low.

Why it's included

Uncertainty-first branch of detection — enables explicit abstain/escalate routing, valuable for high-stakes medical/finance LLM deployments.

Addresses failure modes

These tags are the catalog's claim about what uqlm is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran.

Records evidence of

uqlm can record evidence of these failures (traces, or checks you configure yourself), but it does not detect or prevent them on its own. They are not counted as failure modes it addresses.

Techniques

black-box-uqwhite-box-uqsemantic-uqabstain-routing

View raw YAML on GitHub →