Repository Public
1# Evaluation best practices23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures. Evaluations (**evals**) are a way to test your AI system despite this variability.67This guide provides high-level guidance on designing evals. To get started with the [Evals API](https://developers.openai.com/api/reference/resources/evals), see [evaluating model performance](https://developers.openai.com/api/docs/guides/evals).89OpenAI is deprecating the Evals platform. Existing evals content remains10 available during the transition window. Evals will become read-only for11 existing users on October 31, 2026, and the platform is scheduled to shut down12 on November 30, 2026. See the [deprecations13 page](https://developers.openai.com/api/docs/deprecations#2026-06-03-evals-platform) for the current14 timeline.1516## What are evals?1718Evals are structured tests for measuring a model's performance. They help ensure accuracy, performance, and reliability, despite the nondeterministic nature of AI systems. They're also one of the only ways to _improve_ performance of an LLM-based application (through [fine-tuning](https://developers.openai.com/api/docs/guides/model-optimization)).1920### Types of evals2122When you see the word "evals," it could refer to a few things:2324- Industry benchmarks for comparing models in isolation, like [MMLU](https://github.com/openai/evals/blob/main/examples/mmlu.ipynb) and those listed on [HuggingFace's leaderboard](https://huggingface.co/collections/open-llm-leaderboard/the-big-benchmarks-collection-64faca6335a7fc7d4ffe974a)25- Standard numerical scores—like [ROUGE](https://aclanthology.org/W04-1013/), [BERTScore](https://arxiv.org/abs/1904.09675)—that you can use as you design evals for your use case26- Specific tests you implement to measure your LLM application's performance2728This guide is about the third type: designing your own evals.2930### How to read evals3132You'll often see numerical eval scores between 0 and 1. There's more to evals than just scores. Combine metrics with human judgment to ensure you're answering the right questions.3334**Evals tips**353637- Adopt eval-driven development: Evaluate early and often. Write scoped tests at every stage.38- Design task-specific evals: Make tests reflect model capability in real-world distributions.39- Log everything: Log as you develop so you can mine your logs for good eval cases.40- Automate when possible: Structure evaluations to allow for automated scoring.41- It's a journey, not a destination: Evaluation is a continuous process.42- Maintain agreement: Use human feedback to calibrate automated scoring.4344**Anti-patterns**45464748- Overly generic metrics: Relying solely on academic metrics like perplexity or BLEU score.49- Biased design: Creating eval datasets that don't faithfully reproduce production traffic patterns.50- Vibe-based evals: Using "it seems like it's working" as an evaluation strategy, or waiting until you ship before implementing any evals.51- Ignoring human feedback: Not calibrating your automated metrics against human evals.5253## Design your eval process5455There are a few important components of an eval workflow:56571. **Define eval objective**. What's the success criteria for the eval?581. **Collect dataset**. Which data will help you evaluate against your objective? Consider synthetic eval data, domain-specific eval data, purchased eval data, human-curated eval data, production data, and historical data.591. **Define eval metrics**. How will you check that the success criteria are met?601. **Run and compare evals**. Iterate and improve model performance for your task or system.611. **Continuously evaluate**. Set up continuous evaluation (CE) to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time.6263Let's run through a few examples.6465### Example: Summarizing transcripts6667To test your LLM-based application's ability to summarize transcripts, your eval design might be:68691. **Define eval objective**7071 The model should be able to compete with reference summaries for relevance and accuracy.721. **Collect dataset**7374 Use a mix of production data (collected from user feedback on generated summaries) and datasets created by domain experts (writers) to determine a "good" summary.751. **Define eval metrics**7677 On a held-out set of 1000 reference transcripts → summaries, the implementation should achieve a ROUGE-L score of at least 0.40 and coherence score of at least 80% using G-Eval.781. **Run and compare evals**7980 Use the [Evals API](https://developers.openai.com/api/docs/guides/evals) to create and run evals in the OpenAI dashboard.811. **Continuously evaluate**8283 Set up continuous evaluation (CE) to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time.8485LLMs are better at discriminating between options. Therefore, evaluations86 should focus on tasks like pairwise comparisons, classification, or scoring87 against specific criteria instead of open-ended generation. Aligning88 evaluation methods with LLMs' strengths in comparison leads to more reliable89 assessments of LLM outputs or model comparisons.9091### Example: Q&A over docs9293To test your LLM-based application's ability to do Q&A over docs, your eval design might be:94951. **Define eval objective**9697 The model should be able to provide precise answers, recall context as needed to reason through user prompts, and provide an answer that satisfies the user's need.981. **Collect dataset**99100 Use a mix of production data (collected from users' satisfaction with answers provided to their questions), hard-coded correct answers to questions created by domain experts, and historical data from logs.1011. **Define eval metrics**102103 Context recall of at least 0.85, context precision of over 0.7, and 70+% positively rated answers.1041. **Run and compare evals**105106 Use the [Evals API](https://developers.openai.com/api/docs/guides/evals) to create and run evals in the OpenAI dashboard.1071. **Continuously evaluate**108109 Set up continuous evaluation (CE) to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time.110111When creating an eval dataset, 112 [`gpt-6-astra`](https://developers.openai.com/api/docs/models/gpt-6-astra) 113 is useful for collecting eval examples and edge cases. Consider using it to114 help you generate a diverse set of test data across various scenarios. Ensure115 your test data includes typical cases, edge cases, and adversarial cases. Use116 human expert labellers.117118## Identify where you need evals119120Complexity increases as you move from simple to more complex architectures. Here are four common architecture patterns:121122- [Single-turn model interactions](#single-turn-model-interactions)123- [Workflows](#workflow-architectures)124- [Single-agent](#single-agent-architectures)125- [Multi-agent](#multi-agent-architectures)126127Read about each architecture below to identify where nondeterminism enters your system. That's where you'll want to implement evals.128129### Single-turn model interactions130131In this kind of architecture, the user provides input to the model, and the model processes these inputs (along with any developer prompts provided) to generate a corresponding output.132133#### Example134135As an example, consider an online retail scenario. Your system prompt instructs the model to **categorize the customer's question** into one of the following:136137- `order_status`138- `return_policy`139- `technical_issue`140- `cancel_order`141- `other`142143To ensure a consistent, efficient user experience, the model should **only return the label that matches user intent**. Let's say the customer asks, "What's the status of my order?"144145<table>146 <tr>147 <th>Nondeterminism introduced</th>148 <th>Corresponding area to evaluate</th>149 <th>Example eval questions</th>150 </tr>151 <tr>152 <td>Inputs provided by the developer and user</td>153 <td>154 **Instruction following**: Does the model accurately understand and act155 according to the provided instructions?156 157158 159160 **Instruction following**: Does the model prioritize the system prompt161 over a conflicting user prompt?162 </td>163 <td>164 Does the model stay focused on the triage task or get swayed by the user's165 question?166 </td>167 </tr>168 <tr>169 <td>Outputs generated by the model</td>170 <td>171 **Functional correctness**: Are the model's outputs accurate, relevant,172 and thorough enough to fulfill the intended task or objective?173 </td>174 <td>175 Does the model's determination of intent correctly match the expected176 intent?177 </td>178 </tr>179</table>180181### Workflow architectures182183As you look to solve more complex problems, you'll likely transition from a single-turn model interaction to a multistep workflow that chains together several model calls. Workflows don't introduce any new elements of nondeterminism, but they involve multiple underlying model interactions, which you can evaluate in isolation.184185#### Example186187Take the same example as before, where the customer asks about their order status. A workflow architecture triages the customer request and routes it through a step-by-step process:1881891. Extracting an Order ID1901. Looking up the order details1911. Providing the order details to a model for a final response192193Each step in this workflow has its own system prompt that the model must follow, putting all fetched data into a friendly output.194195<table>196 <tr>197 <th>Nondeterminism introduced</th>198 <th>Corresponding area to evaluate</th>199 <th>Example eval questions</th>200 </tr>201 <tr>202 <td>Inputs provided by the developer and user</td>203 <td>204 **Instruction following**: Does the model accurately understand and act205 according to the provided instructions?206 207208 209210 **Instruction following**: Does the model prioritize the system prompt211 over a conflicting user prompt?212 </td>213 <td>214 Does the model stay focused on the triage task or get swayed by the user's215 question?216 217218 219 Does the model follow instructions to attempt to extract an Order220 ID?221 222223 224225 Does the final response include the order status, estimated arrival date,226 and tracking number?227 </td>228 </tr>229 <tr>230 <td>Outputs generated by the model</td>231 <td>232 **Functional correctness**: Are the model's outputs are accurate,233 relevant, and thorough enough to fulfill the intended task or objective?234 </td>235 <td>236 Does the model's determination of intent correctly match the expected237 intent?238 239240 241242 Does the final response have the correct order status, estimated arrival243 date, and tracking number?244 </td>245 </tr>246</table>247248### Single-agent architectures249250Unlike workflows, agents solve unstructured problems that require flexible decision making. An agent has instructions and a set of tools and dynamically selects which tool to use. This introduces a new opportunity for nondeterminism.251252Tools are developer defined chunks of code that the model can execute. This253 can range from small helper functions to API calls for existing services. For254 example, `check_order_status(order_id)` could be a tool, where it takes the255 argument `order_id` and calls an API to check the order status.256257#### Example258259Let's adapt our customer service example to use a single agent. The agent has access to three distinct tools:260261- Order lookup tool262- Password reset tool263- Product FAQ tool264265When the customer asks about their order status, the agent dynamically decides to either invoke a tool or respond to the customer. For example, if the customer asks, "What is my order status?" the agent can now follow up by requesting the order ID from the customer. This helps create a more natural user experience.266267<table>268 <tr>269 <th>Nondeterminism</th>270 <th>Corresponding area to evaluate</th>271 <th>Example eval questions</th>272 </tr>273 <tr>274 <td>Inputs provided by the developer and user</td>275 <td>276 **Instruction following**: Does the model accurately understand and act277 according to the provided instructions?278 279280 281282 **Instruction following**: Does the model prioritize the system prompt283 over a conflicting user prompt?284 </td>285 <td>286 Does the model stay focused on the triage task or get swayed by the user's287 question?288 289290 291292 Does the model follow instructions to attempt to extract an Order ID?293 </td>294 </tr>295 <tr>296 <td>Outputs generated by the model</td>297 <td>298 **Functional correctness**: Are the model's outputs are accurate,299 relevant, and thorough enough to fulfill the intended task or objective?300 </td>301 <td>302 Does the model's determination of intent correctly match the expected303 intent?304 </td>305 </tr>306 <tr>307 <td>Tools chosen by the model</td>308 <td>309 **Tool selection**: Evaluations that test whether the agent is able to310 select the correct tool to use.311 312313 314315 **Data precision**: Evaluations that verify the agent calls the tool with316 the correct arguments. Typically these arguments are extracted from the317 conversation history, so the goal is to validate this extraction was318 correct.319 </td>320 <td>321 When the user asks about their order status, does the model correctly322 recommend invoking the order lookup tool?323 324325 326327 Does the model correctly extract the user-provided order ID to the lookup328 tool?329 </td>330 </tr>331</table>332333### Multi-agent architectures334335As you add tools and tasks to your single-agent architecture, the model may struggle to follow instructions or select the correct tool to call. Multi-agent architectures help by creating several distinct agents who specialize in different areas. This triaging and handoff among multiple agents introduces a new opportunity for nondeterminism.336337The decision to use a multi-agent architecture should be driven by your evals.338 Starting with a multi-agent architecture adds unnecessary complexity that can339 slow down your time to production.340341#### Example342343Splitting the single-agent example into a multi-agent architecture, we'll have four distinct agents:3443451. Triage agent3461. Order agent3471. Account management agent3481. Sales agent349350When the customer asks about their order status, the triage agent may hand off the conversation to the order agent to look up the order. If the customer changes the topic to ask about a product, the order agent should hand the request back to the triage agent, who then hands off to the sales agent to fetch product information.351352<table>353 <tr>354 <th>Nondeterminism</th>355 <th>Corresponding area to evaluate</th>356 <th>Example eval questions</th>357 </tr>358 <tr>359 <td>Inputs provided by the developer and user</td>360 <td>**Instruction following**: Does the model accurately understand and act according to the provided instructions?361362**Instruction following**: Does the model prioritize the system prompt over a conflicting user prompt?</td>363 <td>Does the model stay focused on the triage task or get swayed by the user's question?364365Assuming the `lookup_order` call returned, does the order agent return a tracking number and delivery date (doesn't have to be the correct one)?</td>366 </tr>367 <tr>368 <td>Outputs generated by the model</td>369 <td>**Functional correctness**: Are the model's outputs are accurate, relevant, and thorough enough to fulfill the intended task or objective?</td>370 <td>Does the model's determination of intent correctly match the expected intent?371372Assuming the `lookup_order` call returned, does the order agent provide the correct tracking number and delivery date in its response?373374Does the order agent follow system instructions to ask the customer their reason for requesting a return before processing the return?</td>375 </tr>376 <tr>377 <td>Tools chosen by the model</td>378 <td>**Tool selection**: Evaluations that test whether the agent is able to select the correct tool to use.379380**Data precision**: Evaluations that verify the agent calls the tool with the correct arguments. Typically these arguments are extracted from the conversation history, so the goal is to validate this extraction was correct.</td>381 <td>Does the order agent correctly call the lookup order tool?382383Does the order agent correctly call the `refund_order` tool?384385Does the order agent call the lookup order tool with the correct order ID?386387Does the account agent correctly call the `reset_password` tool with the correct account ID?</td>388 </tr>389390 <tr>391 <td>Agent handoff</td>392 <td>**Agent handoff accuracy**: Evaluations that test whether each agent can appropriately recognize the decision boundary for triaging to another agent</td>393 <td>When a user asks about order status, does the triage agent correctly pass to the order agent?394395When the user changes the subject to talk about the latest product, does the order agent hand back control to the triage agent?</td>396 </tr>397</table>398399## Create and combine different types of evaluators400401As you design your own evals, there are several specific evaluator types to choose from. Another way to think about this is what role you want the evaluator to play.402403### Metric-based evals404405Quantitative evals provide a numerical score you can use to filter and rank results. They provide useful benchmarks for automated regression testing.406407- **Examples**: Exact match, string match, ROUGE/BLEU scoring, function call accuracy, executable evals (executed to assess functionality or behavior—e.g., text2sql)408- **Challenges**: May not be tailored to specific use cases, may miss nuance409410### Human evals411412Human judgment evals provide the highest quality but are slow and expensive.413414- **Examples**: Skim over system outputs to get a sense of whether they look better or worse; create a randomized, blinded test in which employees, contractors, or outsourced labeling agencies judge the quality of system outputs (e.g., ranking a small set of possible outputs, or giving each a grade of 1-5)415- **Challenges**: Disagreement among human experts, expensive, slow416- **Recommendations**:417 - Conduct multiple rounds of detailed human review to refine the scorecard418 - Implement a "show rather than tell" policy by providing examples of different score levels (e.g., 1, 3, and 8 out of 10)419 - Include a pass/fail threshold in addition to the numerical score420 - A simple way to aggregate multiple reviewers is to take consensus votes421422### LLM-as-a-judge and model graders423424Using models to judge output is cheaper to run and more scalable than human evaluation. Start with [`gpt-6-astra`](https://developers.openai.com/api/docs/models/gpt-6-astra) when you need a strong LLM judge, then validate agreement against your human labels before optimizing for cost or latency.425426- **Examples**:427 - Pairwise comparison: Present the judge model with two responses and ask it to determine which one is better based on specific criteria428 - Single answer grading: The judge model evaluates a single response in isolation, assigning a score or rating based on predefined quality metrics429 - Reference-guided grading: Provide the judge model with a reference or "gold standard" answer, which it uses as a benchmark to evaluate the given response430- **Challenges**: Position bias (response order), verbosity bias (preferring longer responses)431- **Recommendations**:432 - Use pairwise comparison or pass/fail for more reliability433 - Use the most capable model to grade if you can. Start with [`gpt-6-astra`](https://developers.openai.com/api/docs/models/gpt-6-astra), then validate whether a specialized reasoning model performs better for your rubric or reference-answer set434 - Control for response lengths as LLMs bias towards longer responses in general435 - Add reasoning and chain-of-thought as reasoning before scoring improves eval performance436 - Once the LLM judge reaches a point where it's faster, cheaper, and consistently agrees with human annotations, scale up437 - Structure questions to allow for automated grading while maintaining the integrity of the task—a common approach is to reformat questions into multiple choice formats438 - Ensure eval rubrics are clear and detailed439440No strategy is perfect. The quality of LLM-as-Judge varies depending on problem context while using expert human annotators to provide ground-truth labels is expensive and time-consuming.441442## Handle edge cases443444While your evaluations should cover primary, happy-path scenarios for each architecture, real-world AI systems frequently encounter edge cases that challenge system performance. Evaluating these edge cases is important for ensuring reliability and a good user experience.445446We see these edge cases fall into a few buckets:447448### Input variability449450Because users provide input to the model, our system must be flexible to handle the different ways our users may interact, like:451452- Non-English or multilingual inputs453- Formats other than input text (e.g., XML, JSON, Markdown, CSV)454- Input modalities (e.g., images)455456Your evals for instruction following and functional correctness need to accommodate inputs that users might try.457458### Contextual complexity459460Many LLM-based applications fail due to poor understanding of the context of the request. This context could be from the user or noise in the past conversation history.461462Examples include:463464- Multiple questions or intents in a single request465- Typos and misspellings466- Short requests with minimal context (e.g., if a user just says: "returns")467- Long context or long-running conversations468- Tool calls that return data with ambiguous property names (e.g., `"on: 123"`, where "on" is the order number)469- Multiple tool calls, sometimes leading to incorrect arguments470- Multiple agent handoffs, sometimes leading to circular handoffs471472### Personalization and customization473474While AI improves UX by adapting to user-specific requests, this flexibility introduces many edge cases. Clearly define evals for use cases you want to specifically support and block:475476- Jailbreak attempts to get the model to do something different477- Formatting requests (e.g., format as JSON, or use bullet points)478- Cases where user prompts conflict with your system prompts479480## Use evals to improve performance481482When your evals reach a level of maturity that consistently measures performance, shift to using your evals data to improve your application's performance.483484Learn more about [reinforcement fine-tuning](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning) to create a data flywheel.485486## Other resources487488For more inspiration, visit the [OpenAI Cookbook](https://developers.openai.com/cookbook), which contains example code and links to third-party resources, or learn more about our tools for evals:489490- [Evaluating model performance](https://developers.openai.com/api/docs/guides/evals)491- [How to evaluate a summarization task](https://developers.openai.com/cookbook/examples/evaluation/how_to_eval_abstractive_summarization)492- [Fine-tuning](https://developers.openai.com/api/docs/guides/model-optimization)493- [Graders](https://developers.openai.com/api/docs/guides/graders)494- [Evals API reference](https://developers.openai.com/api/reference/resources/evals)