Skip to main content

Tools

Arize Phoenix

12,368

Open-source LLM tracing and evaluation platform built on OpenTelemetry.

Detection Post-generation Fully autonomous Hybrid Stable Activity band B Environment
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: B (raw score 6.64). Shown band: B .

Facets: documented_in=0evaluated_on=2integration_surfaces=1paper_citations=0recent_push=2stars=1.64

Source of truth: pipeline/maturity/score.py

Measured on SAICA-Bench in this configuration

Each row is one run of Arize Phoenix, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good Arize Phoenix is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Set Configuration and run card J (95% interval) TPR FPR No verdict Cost per 1,000 reviews
False 'done' claims
failure mode: Incomplete execution
Real agent output
Arize Phoenix CompletenessEvaluator (deepseek-v4-flash)
Catalog tool · Hybrid · own check · judge model deepseek-v4-flash
Rank 5 of 10 ranked rows; no reviewer reaches J 25% on this set.
Version
arize-phoenix-evals 3.10.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
issue text; agent closing message; final patch (unified diff)
Counts as flagged when
Flagged when Phoenix's completeness evaluator labels the answer 'incomplete'.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
1,200; failed 10 (0.8%): blocked 7, no verdict 3
Samples
600 bad, 600 clean
Answers with no verdict
10 (6 bad, 4 clean): blocked 7, no verdict 3
Cost
$0.82 per 1,000 reviews (api); $0.99 spent on this run. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • CompletenessEvaluator is built for conversations; it was given the issue as the user turn and the closing message plus the final patch as the assistant turn
  • temperature and output budget left to Phoenix's defaults
+6.6%
+3.2% to +10.0%
13.1% 6.5% 10 $0.82

An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.

Description

Open-source LLM and agent observability platform from Arize. Ingests OpenInference/OpenTelemetry traces from any framework, provides a local notebook-friendly UI for span-level inspection, and ships built-in LLM-as-judge evaluators (hallucination, relevance, QA correctness, toxicity, agent-tool-call correctness). Integrates with LangChain, LlamaIndex, CrewAI, AutoGen, and the OpenAI/Anthropic SDKs. Self-hostable container plus an Arize AX managed offering sharing the same data model.

Why it's included

Occupies detection × post_generation × fully_autonomous with a tight OpenTelemetry / OpenInference standardization posture and a notebook-friendly local mode. Worth modeling as a distinct node because it is the de-facto reference implementation of OpenInference semantic conventions for LLM traces.

Security notes

Core project is source-available under Elastic-2.0 and self-hostable via Docker, which makes it a data-residency-friendly observability option. Prompts and completions are the primary payload — redact secrets and PII via the SDK hooks before ingest. OpenInference instrumentation is framework-agnostic; audit auto-instrumentation patches for any that wrap sensitive call sites.

Addresses failure modes

These tags are the catalog's claim about what Arize Phoenix is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran.

  • Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.

Records evidence of

Arize Phoenix can record evidence of these failures (traces, or checks you configure yourself), but it does not detect or prevent them on its own. They are not counted as failure modes it addresses.

Techniques

trace-captureopeninference-instrumentationllm-as-judge-evaldataset-regression-evalspan-annotation

View raw YAML on GitHub →