Vendor: promptfoo
Promptfoo is the industry-leading CLI toolkit for LLM prompt engineering, agent skill evaluation, and automated red teaming, offering matrix testing, semantic assertions, and native CI/CD quality gates.
| Repository | promptfoo/promptfoo |
|---|---|
| GitHub Stars | ★ 24.6k |
| Forks | 2.2k forks |
| Primary Language | TypeScript |
| License | MIT |
| Technical Domain | TOOLING |
$ bash
npm install -g promptfoo
# or with Homebrew
brew install promptfoo
Promptfoo (promptfoo/promptfoo) is the industry-leading open-source testing, evaluation, and red-teaming framework for LLMs and AI agents, created by Ian Webster. When deploying LLM-powered applications to production, developers face a critical void in continuous integration: tweaking a system prompt, switching model versions, or modifying temperature parameters often introduces silent regressions or security vulnerabilities. Promptfoo establishes the standard paradigm for 'Prompt Test-Driven Development' (Prompt TDD).
Architecturally, Promptfoo utilizes a declarative configuration model (YAML/JSON) that evaluates multi-dimensional matrices combining candidate prompts, diverse model providers, and test fixtures. It features dozens of built-in assertion types, ranging from deterministic string matching, regex, and JSON schema validation to custom Python/JS hooks and LLM-as-a-Judge semantic similarity and factual consistency metrics.
For agent security and automated red teaming, Promptfoo includes an automated vulnerability scanner that subjects agent workflows to hundreds of adversarial attack vectors—including prompt injection, jailbreaks, PII exfiltration, harmful content probing, and algorithmic bias. It generates visual matrix comparison reports locally and integrates directly into CI/CD pipelines as quality gates.
Promptfoo was developed at the inflection point where generative AI engineering transitioned from ad-hoc trial-and-error to systematic software engineering. In the early days, engineers tweaked prompts interactively in playground UIs, deploying changes as long as a few sample outputs looked reasonable. However, the probabilistic nature of LLMs meant that subtle modifications frequently caused silent regressions in untested edge cases.
Ian Webster translated the battle-tested principles of unit testing frameworks (such as Jest and Pytest) into the LLM ecosystem. Promptfoo emphasizes combining deterministic assertions with parametrized test fixtures. Through a lightweight, local-first CLI with zero telemetry risk, Promptfoo enables developers to verify prompt changes in seconds with quantitative pass-rate and cost matrices.
The core testing philosophy centers on tiered assertions: utilizing cheap string and regex checks to catch 80% of syntax defects, schema assertions for tool validation, and reserving costly LLM-as-a-Judge evaluations for high-level semantic rubrics.
In prompt optimization and model evaluation, teams benchmark Claude 3.7 Sonnet, GPT-4o, and DeepSeek across 50+ domain prompts to analyze quality, latency, and cost trade-offs with empirical data.
In continuous integration pipelines, Promptfoo runs as a GitHub Actions quality gate, automatically blocking pull requests if prompt modifications reduce assertion pass rates below target thresholds.
In security auditing, executing promptfoo redteam autonomously launches adversarial jailbreak, prompt injection, and PII exfiltration attacks to evaluate an agent's guardrails.
In enterprise RAG pipelines, developers combine contains-json, levenshtein, and llm-rubric assertions to quantify retrieval relevance and response factuality automatically.
Install Promptfoo globally via npm or Homebrew:
npm install -g promptfoo
# or with Homebrew
brew install promptfooInitialize a new evaluation configuration:
promptfoo initDefine candidate prompts, model providers, and assertion tests in promptfooconfig.yaml:
prompts:
- 'You are a senior engineer. Explain the core value of {{topic}} in one sentence.'
- 'You are a technical mentor. Explain {{topic}} to a beginner simply.'
providers:
- openai:gpt-4o-mini
- anthropic:claude-3-5-haiku-20241022
tests:
- vars:
topic: Docker containers
assert:
- type: contains
value: isolation
- type: llm-rubric
value: Explanation must be technically accurate and under 50 words
- vars:
topic: Vector databases
assert:
- type: similar
value: Specialized database optimized for indexing and querying high-dimensional vector embeddings
threshold: 0.75Run the evaluation matrix and view the local web dashboard:
promptfoo eval
promptfoo viewIn privacy and production architecture, Promptfoo operates strictly local-first. All prompts, test inputs, and evaluation metrics remain on the local machine or private CI runners without external telemetry, aligning fully with strict enterprise data privacy standards.
Regarding performance, Promptfoo supports high-throughput asynchronous execution (tunable via --concurrency), completing matrix runs with thousands of evaluations in minutes. Custom Python and JavaScript hooks allow seamless extension of custom providers and assertion metrics.
Cost transparency is built-in: each test run prints exact token counts and estimated USD costs per provider, helping engineering teams budget evaluation workloads effectively.
With over 24,000 GitHub stars, Promptfoo is widely regarded as one of the gold standards for LLM testing and red teaming, deployed across engineering organizations at AWS, Shopify, Discord, and Databricks.
A prominent edtech enterprise deployed Promptfoo across 200 tutoring prompts, catching 14 regression and jailbreak defects prior to production releases and protecting end-user safety.
As enterprise AI compliance and security red teaming standards become mandatory globally, Promptfoo's open-source automated vulnerability scanner positions it as essential infrastructure for LLMOps security governance.
Promptfoo - LLM Prompt Testing, Agent Skill Evaluation & Red Teaming Suite is an open-source AI project developed primarily in TypeScript under the MIT license. Promptfoo is the industry-leading CLI toolkit for LLM prompt engineering, agent skill evaluation, and automated red teaming, offering matrix testing, semantic assertions, and native CI/CD quality gates.. Promptfoo was developed at the inflection point where generative AI engineering transitioned from ad-hoc trial-and-error to systematic software engineering. In the early days, engineers tweaked prompts interactively in playground UIs, deploying changes as long as a few sample outputs looked reasonable. However, the probabilistic nature of LLMs meant that subtle modifications frequently caused silent regressions in untested edge cases. Ian Webster translated the battle-tested principles of unit testing frameworks (such as Jest and Pytest) into the LLM ecosystem. Promptfoo emphasizes combining deterministic assertions with parametrized test fixtures. Through a lightweight, local-first CLI with zero telemetry risk, Promptfoo enables developers to verify prompt changes in seconds with quantitative pass-rate and cost matrices. The core testing philosophy centers on tiered assertions: utilizing cheap string and regex checks to catch 80% of syntax defects, schema assertions for tool validation, and reserving costly LLM-as-a-Judge evaluations for high-level semantic rubrics.
Install Promptfoo globally via npm or Homebrew:
npm install -g promptfoo
# or with Homebrew
brew install promptfooInitialize a new evaluation configuration:
promptfoo initDefine candidate prompts, model providers, and assertion tests in promptfooconfig.yaml:
prompts:
- 'You are a senior engineer. Explain the core value of {{topic}} in one sentence.'
- 'You are a technical mentor. Explain {{topic}} to a beginner simply.'
providers:
- openai:gpt-4o-mini
- anthropic:claude-3-5-haiku-20241022
tests:
- vars:
topic: Docker containers
assert:
- type: contains
value: isolation
- type: llm-rubric
value: Explanation must be technically accurate and under 50 words
- vars:
topic: Vector databases
assert:
- type: similar
value: Specialized database optimized for indexing and querying high-dimensional vector embeddings
threshold: 0.75Run the evaluation matrix and view the local web dashboard:
promptfoo eval
promptfoo viewPromptfoo - LLM Prompt Testing, Agent Skill Evaluation & Red Teaming Suite is well-suited for Prompt Optimization & Multi-Model Benchmarking, Pre-Deployment Automated Red Teaming, CI/CD Continuous Integration Quality Gates, Enterprise RAG Quality & Factuality Evaluation. With an overall rating of 4.9/5, it offers strong community activity, reliable performance, and easy integration with existing AI pipelines.
In privacy and production architecture, Promptfoo operates strictly local-first. All prompts, test inputs, and evaluation metrics remain on the local machine or private CI runners without external telemetry, aligning fully with strict enterprise data privacy standards. Regarding performance, Promptfoo supports high-throughput asynchronous execution (tunable via --concurrency), completing matrix runs with thousands of evaluations in minutes. Custom Python and JavaScript hooks allow seamless extension of custom providers and assertion metrics. Cost transparency is built-in: each test run prints exact token counts and estimated USD costs per provider, helping engineering teams budget evaluation workloads effectively.
Minimal tool for running large language models locally