confident-ai/deepeval

Project overview

An open-source evaluation framework, modeled after Pytest, for unit testing and evaluating large language model applications. It provides ready-to-use metrics to help development teams assess LLM quality, compare model outputs, and detect prompt drifting.

Project type

Evaluation & Observability

Use cases

Knowledge Q&A

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Developers, AI engineers, and data teams who need to prevent prompt drifting and evaluate LLM systems programmatically.
• Teams looking to integrate LLM unit testing into a Python-based CI/CD workflow.

Key capabilities

• Manually instrument or use framework integrations to run datasets through an app with full traceability.
• Guardrails for LLM applications.
• DeepEval's graph-based deterministic LLM-as-a-judge metric builder.

Limitations and risks

• Requires Python 3.9 or higher.
• Requires an OpenAI API key or a custom LLM for LLM-as-a-judge evaluation.
• Requires writing Python code to define and manage test cases.

Getting started

• The setup requires Python knowledge and writing test cases in code. To begin, install deepeval via pip, create a Python test file with an LLMTestCase and metric, set an LLM API key as an environment variable, and run tests using the CLI.

Evidence and sources

• README: DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps.
• Release: Python 4.1.0
• README: Large variety of ready-to-use LLM eval metrics (all with explanations) powered by **ANY** LLM of your choice, statistical methods, or NLP models that run **locally on your machine…
• README: Use `evals_iterator()` to run the same dataset through your app, whether you instrument it manually or through one of DeepEval's framework integrations.
• README: - [ ] DAG custom metrics - [ ] Guardrails

Reviewed at: 2026-07-17T13:51:15Z