DeepEval is an open-source evaluation framework that applies a Pytest-like workflow to unit testing and regression testing of LLM applications. It provides ready-to-use metrics for measuring output quality and detecting prompt drift, requiring an LLM such as OpenAI or a custom model to serve as the judge for evaluations.
Project overview
The project addresses the challenge of preventing prompt regression by adapting the familiar Pytest unit-testing model to LLM outputs, allowing developers to run automated test suites that score model responses on a 0-to-1 scale.
Project type
Evaluation & Observability · Prompt Engineering
Use cases
Knowledge Q&A
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers and AI engineers who need to evaluate LLM outputs, prevent prompt drifting, and compare model performance using a Pytest-like testing structure.
Key capabilities
Manually instrument or use framework integrations to run datasets through an application with full traceability.
Guardrails for LLM applications.
A graph-based deterministic LLM-as-a-judge metric builder.
Limitations and risks
Requires Python 3.9 or higher.
An OpenAI API key or a custom LLM is required to perform LLM-as-a-judge evaluations.
The framework can connect to Confident AI, an optional external platform for managing datasets, tracing, and running evaluations, and telemetry is optional.
Getting started
Installation and setup require Python 3.9 or higher and Python knowledge. The path to first success involves installing DeepEval via pip, creating a Python test file with an LLMTestCase and metric, setting an LLM API key as an environment variable, and running the test via the CLI.
Evidence and sources
README: DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps.
Release: Python 4.1.0
README: Large variety of ready-to-use LLM eval metrics (all with explanations) powered by **ANY** LLM of your choice, statistical methods, or NLP models that run **locally on your machine…
README: Use `evals_iterator()` to run the same dataset through your app, whether you instrument it manually or through one of DeepEval's framework integrations.