promptfoo/promptfoo

Project overview

promptfoo is a CLI and library for declaratively testing prompts, models, and RAG systems, and for red-teaming LLM applications to identify vulnerabilities. Evals run locally so prompt data stays on your machine, while integration with external LLM providers is optional but required for provider-specific testing.

Project type

Evaluation & Observability · Infrastructure

Deployment

Refer to project documentation

License

MIT

Best for

• Developers and AI engineers who need a CLI or library to automate prompts, model evaluations, and RAG testing locally.
• Operations teams requiring automated vulnerability scanning and CI/CD integration for LLM applications.

Key capabilities

• Test prompts and models using automated evaluations to generate output matrices.
• Secure LLM applications by running red teaming exercises and vulnerability scans.
• Automate checks and gates within CI/CD pipelines.
• Review pull requests for LLM-related security and compliance issues.
• View evaluation results locally and share them with your team.

Limitations and risks

• Requires Node.js ^20.20.0 or >=22.22.0 for npm and npx usage. Node.js 20 support ends July 30, 2026.
• Most external LLM providers require an API key to perform evaluations or vulnerability scans.

Getting started

• Install globally via npm, initialize a project with examples, navigate to the directory, execute an evaluation, and view the results.

Evidence and sources

• README: <code>promptfoo</code> is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.
• README: Requires [Node.js](https://nodejs.org/en/download) `^20.20.0` or `>=22.22.0` for npm and npx usage.
• README: - **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)
• README: - **Developer-first**: Fast, with features like live reload and caching
• README: Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed.

Reviewed at: 2026-07-17T11:01:21Z