Verified project record

promptfoo/promptfoo

promptfoo is a developer-first library and CLI for declarative testing, automated evaluation, and vulnerability scanning of LLM prompts, agents, and retrieval pipelines. It supports local orchestration and integrates with multiple LLM providers, including remote APIs and local runtimes.

Project overview

It addresses trial-and-error prompt and RAG testing by providing automated evaluations, red teaming, and CI/CD automation from a developer-first CLI.

Project type: Evaluation & Observability · Infrastructure · Prompt Engineering
Deployment: Refer to project documentation
License: MIT

Best for

Developers, AI engineers, and operations teams who need to evaluate prompts, test models, and automate security checks via a CLI or library.

Key capabilities

Test prompts and models using automated evaluation matrices.
Secure LLM applications by identifying vulnerabilities through red teaming and automated scanning.
Automate evaluation and security checks directly within continuous integration and continuous deployment pipelines.
Review pull requests to detect LLM-related security and compliance issues.
View evaluation outputs locally and share the results with your team.

Limitations and risks

Requires Node.js version 20.20.0 or higher, or version 22.22.0 or higher for npm and npx usage. Node.js 20 support ends on July 30, 2026.
Most external LLM providers require an API key to execute model evaluations and red teaming tasks.

Getting started

Installation is rated easy and supports npm, pip, or brew. A quick start path is available by installing globally via npm, running the init command with an example, navigating to the directory, executing the evaluation, and viewing the results.

Alternatives and comparisons

Validates LLM interactions against specific risks using Input/Output Guards and generates structured data from LLMs.
Provides an open source AI engineering platform with LLM observability, prompt management, and dataset evaluations.

Project comparisons

Evidence and sources

README: <code>promptfoo</code> is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.
README: Requires [Node.js](https://nodejs.org/en/download) `^20.20.0` or `>=22.22.0` for npm and npx usage.
README: - **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)
README: - **Developer-first**: Fast, with features like live reload and caching
README: Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed.

Source updated at: 2026-07-16T18:26:38Z

Open original GitHub project