DeepEval
19,587The open-source LLM evaluation framework for testing and evaluating LLM applications across 20+ custom and researched metrics.
Activity band details
Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.
Computed band: D
(raw score 6.72).
Shown band: D
.
Facets:
documented_in=0evaluated_on=2integration_surfaces=1paper_citations=0recent_push=2stars=1.72
capped from B to D by maturity_status='experimental'
Source of truth: pipeline/maturity/score.py
Measured on SAICA-Bench in this configuration
Each row is one run of DeepEval, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good DeepEval is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
| Set | Configuration and run card | J (95% interval) | TPR | FPR | No verdict | Cost per 1,000 reviews |
|---|---|---|---|---|---|---|
|
False 'done' claims
failure mode: Incomplete execution
Real agent output
|
DeepEval TaskCompletionMetric (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 8 of 10 ranked rows; no reviewer reaches J 25% on this set.
How this run differs from the tool's defaults
|
+0.8%
−0.6% to +2.4%
|
1.9% | 1.0% | 10 | $0.46 |
An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.
Description
DeepEval is an open-source framework for unit testing and evaluating LLM applications. It provides 20+ research-backed evaluation metrics including hallucination detection, correctness, answer relevancy, faithfulness, contextual recall/precision, and prompt injection detection. Developers can integrate it into CI/CD pipelines for continuous evaluation of LLM outputs. Supports custom metrics, chain-of-thought evaluation, and synthetic data generation for test cases.
Why it's included
Occupies detection × post_generation × full_hitl as a unit-testing harness for LLM applications: developers write pytest-style assertions over 20+ research-backed metrics (faithfulness, contextual recall/precision, hallucination, G-Eval chain-of-thought) and gate CI on them. Distinguished from observability-first tools by centering test-case authoring rather than trace capture; it is open-source and Pytest-native. Useful as the canonical CI-gate exemplar for the post-generation evaluation cell.
Addresses failure modes
These tags are the catalog's claim about what DeepEval is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran. Not measured here: Context pollution and Scope creep.
- Context pollution — Trajectory drift, hallucination spirals, overlong or off-task context.
- Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.
- Scope creep — Actions outside declared task boundary.
Records evidence of
DeepEval can record evidence of these failures (traces, or checks you configure yourself), but it does not detect or prevent them on its own. They are not counted as failure modes it addresses.