Repository Public
1# Working with evals23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Evaluations (often called **evals**) test model outputs to ensure they meet style and content criteria that you specify. Writing evals to understand how your LLM applications are performing against your expectations, especially when upgrading or trying new models, is an essential component to building reliable applications.67In this guide, we will focus on **configuring evals programmatically using the [Evals API](https://developers.openai.com/api/reference/resources/evals)**. If you prefer, you can also configure evals [in the OpenAI dashboard](https://platform.openai.com/evaluations).89OpenAI is deprecating the Evals platform. Existing evals content remains10 available during the transition window. Evals will become read-only for11 existing users on October 31, 2026, and the platform is scheduled to shut down12 on November 30, 2026. See the [deprecations13 page](https://developers.openai.com/api/docs/deprecations#2026-06-03-evals-platform) for the current14 timeline.1516If you're new to evaluations, or want a more iterative environment to17 experiment in as you build your eval, consider trying18 [Datasets](https://developers.openai.com/api/docs/guides/evaluation-getting-started) instead.1920Broadly, there are three steps to build and run evals for your LLM application.21221. Describe the task to be done as an eval231. Run your eval with test inputs (a prompt and input data)241. Analyze the results, then iterate and improve on your prompt2526This process is somewhat similar to behavior-driven development (BDD), where you begin by specifying how the system should behave before implementing and testing the system. Let's see how we would complete each of the steps above using the [Evals API](https://developers.openai.com/api/reference/resources/evals).2728## Create an eval for a task2930Creating an eval begins by describing a task to be done by a model. Let's say that we would like to use a model to classify the contents of IT support tickets into one of three categories: `Hardware`, `Software`, or `Other`.3132To implement this use case, you can use either the [Chat Completions API](https://developers.openai.com/api/reference/resources/chat) or the [Responses API](https://developers.openai.com/api/reference/resources/responses). Both examples below combine a [developer message](https://developers.openai.com/api/docs/guides/text) with a user message containing the text of a support ticket.333435 Categorize IT support tickets3637```javascript38import OpenAI from "openai";39const client = new OpenAI();4041const instructions = `42You are an expert in categorizing IT support tickets. Given the support43ticket below, categorize the request into one of "Hardware", "Software",44or "Other". Respond with only one of those words.45`;4647const ticket = "My monitor won't turn on - help!";4849const response = await client.responses.create({50 model: "gpt-6-astra",51 input: [52 { role: "developer", content: instructions },53 { role: "user", content: ticket },54 ],55});5657console.log(response.output_text);58```5960```python61from openai import OpenAI6263client = OpenAI()6465instructions = """66You are an expert in categorizing IT support tickets. Given the support67ticket below, categorize the request into one of "Hardware", "Software",68or "Other". Respond with only one of those words.69"""7071ticket = "My monitor won't turn on - help!"7273response = client.responses.create(74 model="gpt-6-astra",75 input=[76 {"role": "developer", "content": instructions},77 {"role": "user", "content": ticket},78 ],79)8081print(response.output_text)82```8384```go85package main8687import (88 "context"89 "fmt"9091 "github.com/openai/openai-go/v3"92 "github.com/openai/openai-go/v3/responses"93)9495func main() {96 client := openai.NewClient()97 instructions := "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of \"Hardware\", \"Software\", or \"Other\". Respond with only one of those words."98 response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{99 Model: "gpt-6-astra",100 Input: responses.ResponseNewParamsInputUnion{OfInputItemList: responses.ResponseInputParam{101 responses.ResponseInputItemParamOfMessage(instructions, responses.EasyInputMessageRoleDeveloper),102 responses.ResponseInputItemParamOfMessage("My monitor won't turn on - help!", responses.EasyInputMessageRoleUser),103 }},104 })105 if err != nil {106 panic(err)107 }108 fmt.Println(response.OutputText())109}110```111112```java113import com.openai.client.OpenAIClient;114import com.openai.client.okhttp.OpenAIOkHttpClient;115import com.openai.models.responses.EasyInputMessage;116import com.openai.models.responses.ResponseCreateParams;117import com.openai.models.responses.ResponseInputItem;118import java.util.List;119120ResponseCreateParams params =121 ResponseCreateParams.builder()122 .model("gpt-6-astra")123 .inputOfResponse(124 List.of(125 ResponseInputItem.ofEasyInputMessage(126 EasyInputMessage.builder()127 .role(EasyInputMessage.Role.DEVELOPER)128 .content(129 "You are an expert in categorizing IT support tickets. Categorize each request as Hardware, Software, or Other. Respond with only one of those words.")130 .build()),131 ResponseInputItem.ofEasyInputMessage(132 EasyInputMessage.builder()133 .role(EasyInputMessage.Role.USER)134 .content("My monitor won't turn on - help!")135 .build())))136 .build();137138client.responses().create(params).output().stream()139 .flatMap(item -> item.message().stream())140 .flatMap(message -> message.content().stream())141 .flatMap(content -> content.outputText().stream())142 .forEach(text -> System.out.println(text.text()));143```144145```csharp146using OpenAI.Responses;147#pragma warning disable OPENAI001148149string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;150ResponsesClient client = new(key);151152ResponseResult response = await client.CreateResponseAsync(153 "gpt-6-astra",154 [155 ResponseItem.CreateDeveloperMessageItem(156 "Categorize the IT support ticket as Hardware, Software, or Other. Respond with only one of those words."157 ),158 ResponseItem.CreateUserMessageItem("My monitor will not turn on. Help!"),159 ]160);161162Console.WriteLine(response.GetOutputText());163```164165```ruby166require "openai"167168client = OpenAI::Client.new169instructions = <<~INSTRUCTIONS170 You are an expert in categorizing IT support tickets. Given the support171 ticket below, categorize the request as Hardware, Software, or Other.172 Respond with only one of those words.173INSTRUCTIONS174response = client.responses.create(175 model: "gpt-6-astra",176 input: [177 {178 role: :developer,179 content: instructions180 },181 {182 role: :user,183 content: "My monitor won't turn on - help!"184 }185 ]186)187puts(response.output_text)188```189190```bash191curl https://api.openai.com/v1/responses \192 -H "Authorization: Bearer $OPENAI_API_KEY" \193 -H "Content-Type: application/json" \194 -d '{195 "model": "gpt-6-astra",196 "input": [197 {198 "role": "developer",199 "content": "Categorize the following support ticket into one of Hardware, Software, or Other."200 },201 {202 "role": "user",203 "content": "My monitor wont turn on - help!"204 }205 ]206 }'207```208209210211212213Let's set up an eval to test this behavior [via API](https://developers.openai.com/api/reference/resources/evals). An eval needs two key ingredients:214215- `data_source_config`: A schema for the test data you will use along with the eval.216- `testing_criteria`: The [graders](https://developers.openai.com/api/docs/guides/graders) that determine if the model output is correct.217218Create an eval219220```javascript221import OpenAI from "openai";222const openai = new OpenAI();223224const evalObj = await openai.evals.create({225 name: "IT Ticket Categorization",226 data_source_config: {227 type: "custom",228 item_schema: {229 type: "object",230 properties: {231 ticket_text: { type: "string" },232 correct_label: { type: "string" },233 },234 required: ["ticket_text", "correct_label"],235 },236 include_sample_schema: true,237 },238 testing_criteria: [239 {240 type: "string_check",241 name: "Match output to human label",242 input: "{{ sample.output_text }}",243 operation: "eq",244 reference: "{{ item.correct_label }}",245 },246 ],247});248249console.log(evalObj);250```251252```python253from openai import OpenAI254255client = OpenAI()256257eval_obj = client.evals.create(258 name="IT Ticket Categorization",259 data_source_config={260 "type": "custom",261 "item_schema": {262 "type": "object",263 "properties": {264 "ticket_text": {"type": "string"},265 "correct_label": {"type": "string"},266 },267 "required": ["ticket_text", "correct_label"],268 },269 "include_sample_schema": True,270 },271 testing_criteria=[272 {273 "type": "string_check",274 "name": "Match output to human label",275 "input": "{{ sample.output_text }}",276 "operation": "eq",277 "reference": "{{ item.correct_label }}",278 }279 ],280)281282print(eval_obj)283```284285```ruby286require "openai"287288client = OpenAI::Client.new289evaluation = client.evals.create(290 name: "Support answer quality",291 data_source_config: {292 type: :custom,293 item_schema: {294 type: :object,295 properties: { input: { type: :string } },296 required: ["input"]297 }298 },299 testing_criteria: [300 {301 type: :string_check,302 name: "mentions_refund",303 input: "{{sample.output_text}}",304 operation: :contains,305 reference: "refund"306 }307 ]308)309puts(evaluation.id)310```311312```bash313curl https://api.openai.com/v1/evals \314 -H "Authorization: Bearer $OPENAI_API_KEY" \315 -H "Content-Type: application/json" \316 -d '{317 "name": "IT Ticket Categorization",318 "data_source_config": {319 "type": "custom",320 "item_schema": {321 "type": "object",322 "properties": {323 "ticket_text": { "type": "string" },324 "correct_label": { "type": "string" }325 },326 "required": ["ticket_text", "correct_label"]327 },328 "include_sample_schema": true329 },330 "testing_criteria": [331 {332 "type": "string_check",333 "name": "Match output to human label",334 "input": "{{ sample.output_text }}",335 "operation": "eq",336 "reference": "{{ item.correct_label }}"337 }338 ]339 }'340```341342343344345### Explanation: data_source_config parameter346347348349Running this eval will require a test data set that represents the type of data you expect your prompt to work with (more on creating the test data set later in this guide). In our `data_source_config` parameter, we specify that each **item** in the data set will conform to a [JSON schema](https://json-schema.org/) with two properties:350351- `ticket_text`: a string of text with the contents of a support ticket352- `correct_label`: a "ground truth" output that the model should match, provided by a human353354Since we will be referencing a **sample** in our test criteria (the output generated by a model given our prompt), we also set `include_sample_schema` to `true`.355356```json357{358 "type": "custom",359 "item_schema": {360 "type": "object",361 "properties": {362 "ticket": { "type": "string" },363 "category": { "type": "string" }364 },365 "required": ["ticket", "category"]366 },367 "include_sample_schema": true368}369```370371372373374375376377### Explanation: testing_criteria parameter378379380381In our `testing_criteria`, we define how we will conclude if the model output satisfies our requirements for each item in the data set. In this case, we just want the model to output one of three category strings based on the input ticket. The string it outputs should exactly match the human-labeled `correct_label` field in our test data. So in this case, we will want to use a `string_check` grader to evaluate the output.382383In the test configuration, we will introduce template syntax, represented by the `{{` and `}}` brackets below. This is how we will insert dynamic content into the test for this eval.384385- `{{ item.correct_label }}` refers to the ground truth value in our test data.386- `{{ sample.output_text }}` refers to the content we will generate from a model to evaluate our prompt - we'll show how to do that when we actually kick off the eval run.387388```json389{390 "type": "string_check",391 "name": "Category string match",392 "input": "{{ sample.output_text }}",393 "operation": "eq",394 "reference": "{{ item.category }}"395}396```397398399400401402After creating the eval, it will be assigned a UUID that you will need to address it later when kicking off a run.403404```json405{406 "object": "eval",407 "id": "eval_67e321d23b54819096e6bfe140161184",408 "data_source_config": {409 "type": "custom",410 "schema": { ... omitted for brevity... }411 },412 "testing_criteria": [413 {414 "name": "Match output to human label",415 "id": "Match output to human label-c4fdf789-2fa5-407f-8a41-a6f4f9afd482",416 "type": "string_check",417 "input": "{{ sample.output_text }}",418 "reference": "{{ item.correct_label }}",419 "operation": "eq"420 }421 ],422 "name": "IT Ticket Categorization",423 "created_at": 1742938578,424 "metadata": {}425}426```427428Now that we've created an eval that describes the desired behavior of our application, let's test a prompt with a set of test data.429430## Test a prompt with your eval431432Now that we have defined how we want our app to behave in an eval, let's construct a prompt that reliably generates the correct output for a representative sample of test data.433434### Uploading test data435436There are several ways to provide test data for eval runs, but it may be convenient to upload a [JSONL](https://jsonlines.org/) file that contains data in the schema we specified when we created our eval. A sample JSONL file that conforms to the schema we set up is below:437438```json439{ "item": { "ticket_text": "My monitor won't turn on!", "correct_label": "Hardware" } }440{ "item": { "ticket_text": "I'm in vim and I can't quit!", "correct_label": "Software" } }441{ "item": { "ticket_text": "Best restaurants in Cleveland?", "correct_label": "Other" } }442```443444This data set contains both test inputs and ground truth labels to compare model outputs against.445446Next, let's upload our test data file to the OpenAI platform so we can reference it later. You can upload files [in the dashboard here](https://platform.openai.com/storage/files), but it's possible to [upload files via API](https://developers.openai.com/api/reference/resources/files/methods/create) as well. The samples below assume you are running the command in a directory where you saved the sample JSON data above to a file called `tickets.jsonl`:447448Upload a test data file449450```javascript451import fs from "fs";452import OpenAI from "openai";453454const openai = new OpenAI();455456const file = await openai.files.create({457 file: fs.createReadStream("fixtures/tickets.jsonl"),458 purpose: "evals",459});460461console.log(file);462```463464```python465from openai import OpenAI466467client = OpenAI()468469file = client.files.create(file=open("tickets.jsonl", "rb"), purpose="evals")470471print(file)472```473474```go475package main476477import (478 "context"479 "fmt"480 "os"481482 "github.com/openai/openai-go/v3"483)484485func main() {486 client := openai.NewClient()487 file, err := os.Open("tickets.jsonl")488 if err != nil {489 panic(err)490 }491 defer file.Close()492 uploaded, err := client.Files.New(context.Background(), openai.FileNewParams{493 File: file,494 Purpose: openai.FilePurposeEvals,495 })496 if err != nil {497 panic(err)498 }499 fmt.Println(uploaded.ID)500}501```502503```java504import com.openai.client.OpenAIClient;505import com.openai.client.okhttp.OpenAIOkHttpClient;506import com.openai.models.files.FileCreateParams;507import com.openai.models.files.FilePurpose;508import java.nio.file.Path;509510var file =511 client512 .files()513 .create(514 FileCreateParams.builder()515 .file(Path.of(System.getenv("OPENAI_EXAMPLE_FILE_PATH")))516 .purpose(FilePurpose.EVALS)517 .build());518519System.out.println(file.id());520```521522```ruby523require "openai"524require "pathname"525526client = OpenAI::Client.new527file = Pathname("tickets.jsonl")528uploaded = client.files.create(file: file, purpose: :evals)529puts(uploaded.id)530```531532```bash533curl https://api.openai.com/v1/files \534 -H "Authorization: Bearer $OPENAI_API_KEY" \535 -F purpose="evals" \536 -F file="@tickets.jsonl"537```538539540When you upload the file, make note of the unique `id` property in the response payload (also available in the UI if you uploaded via the browser) - we will need to reference that value later:541542```json543{544 "object": "file",545 "id": "file-CwHg45Fo7YXwkWRPUkLNHW",546 "purpose": "evals",547 "filename": "tickets.jsonl",548 "bytes": 208,549 "created_at": 1742834798,550 "expires_at": null,551 "status": "processed",552 "status_details": null553}554```555556### Creating an eval run557558With our test data in place, let's evaluate a prompt and see how it performs against our test criteria. Via API, we can do this by [creating an eval run](https://developers.openai.com/api/reference/resources/evals/methods/create).559560Make sure to replace `YOUR_EVAL_ID` and `YOUR_FILE_ID` with the unique IDs of the eval configuration and test data files you created in the steps above.561562563 Create an eval run564565```javascript566import OpenAI from "openai";567const openai = new OpenAI();568569const run = await openai.evals.runs.create("YOUR_EVAL_ID", {570 name: "Categorization text run",571 data_source: {572 type: "responses",573 model: "gpt-6-astra",574 input_messages: {575 type: "template",576 template: [577 {578 role: "developer",579 content:580 "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.",581 },582 { role: "user", content: "{{ item.ticket_text }}" },583 ],584 },585 source: { type: "file_id", id: "YOUR_FILE_ID" },586 },587});588589console.log(run);590```591592```python593from openai import OpenAI594595client = OpenAI()596597run = client.evals.runs.create(598 "YOUR_EVAL_ID",599 name="Categorization text run",600 data_source={601 "type": "responses",602 "model": "gpt-6-astra",603 "input_messages": {604 "type": "template",605 "template": [606 {607 "role": "developer",608 "content": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of 'Hardware', 'Software', or 'Other'. Respond with only one of those words.",609 },610 {"role": "user", "content": "{{ item.ticket_text }}"},611 ],612 },613 "source": {"type": "file_id", "id": "YOUR_FILE_ID"},614 },615)616617print(run)618```619620```ruby621require "openai"622623client = OpenAI::Client.new624run = client.evals.runs.create(625 "YOUR_EVAL_ID",626 name: "Categorization text run",627 data_source: {628 type: :responses,629 source: {630 type: :file_id,631 id: "YOUR_FILE_ID"632 },633 input_messages: {634 type: :template,635 template: [636 {637 role: :developer,638 content: "Categorize the ticket as Hardware, Software, or Other."639 },640 {641 role: :user,642 content: "{{ item.ticket_text }}"643 }644 ]645 },646 model: "gpt-6-astra"647 }648)649puts(run.id)650```651652```bash653curl https://api.openai.com/v1/evals/YOUR_EVAL_ID/runs \654 -H "Authorization: Bearer $OPENAI_API_KEY" \655 -H "Content-Type: application/json" \656 -d '{657 "name": "Categorization text run",658 "data_source": {659 "type": "responses",660 "model": "gpt-6-astra",661 "input_messages": {662 "type": "template",663 "template": [664 {"role": "developer", "content": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of Hardware, Software, or Other. Respond with only one of those words."},665 {"role": "user", "content": "{{ item.ticket_text }}"}666 ]667 },668 "source": { "type": "file_id", "id": "YOUR_FILE_ID" }669 }670 }'671```672673674675676677When we create the run, we set up a prompt using either a [Chat Completions](https://developers.openai.com/api/docs/guides/text?api-mode=chat) messages array or a [Responses](https://developers.openai.com/api/reference/resources/responses) input. This prompt is used to generate a model response for every line of test data in your data set. We can use the double curly brace syntax to template in the dynamic variable `item.ticket_text`, which is drawn from the current test data item.678679If the eval run is successfully created, you'll receive an API response that looks like this:680681682```json683{684 "object": "eval.run",685 "id": "evalrun_67e44c73eb6481909f79a457749222c7",686 "eval_id": "eval_67e44c5becec81909704be0318146157",687 "report_url": "https://platform.openai.com/evaluation/evals/abc123",688 "status": "queued",689 "model": "gpt-4.1",690 "name": "Categorization text run",691 "created_at": 1743015028,692 "result_counts": { ... },693 "per_model_usage": null,694 "per_testing_criteria_results": null,695 "data_source": {696 "type": "responses",697 "source": {698 "type": "file_id",699 "id": "file-J7MoX9ToHXp2TutMEeYnwj"700 },701 "input_messages": {702 "type": "template",703 "template": [704 {705 "type": "message",706 "role": "developer",707 "content": {708 "type": "input_text",709 "text": "You are an expert in...."710 }711 },712 {713 "type": "message",714 "role": "user",715 "content": {716 "type": "input_text",717 "text": "{{item.ticket_text}}"718 }719 }720 ]721 },722 "model": "gpt-4.1",723 "sampling_params": null724 },725 "error": null,726 "metadata": {}727}728```729730731732733Your eval run has now been queued, and it will execute asynchronously as it processes every row in your data set, generating responses for testing with the prompt and model we specified.734735## Analyze the results736737To receive updates when a run succeeds, fails, or is canceled, create a webhook endpoint and subscribe to the `eval.run.succeeded`, `eval.run.failed`, and `eval.run.canceled` events. See the [webhooks guide](https://developers.openai.com/api/docs/guides/webhooks) for more details.738739Depending on the size of your dataset, the eval run may take some time to complete. You can view current status in the dashboard, but you can also [fetch the current status of an eval run via API](https://developers.openai.com/api/reference/resources/evals/methods/retrieve):740741Retrieve eval run status742743```javascript744import OpenAI from "openai";745const openai = new OpenAI();746747const run = await openai.evals.runs.retrieve("YOUR_RUN_ID", {748 eval_id: "YOUR_EVAL_ID",749});750console.log(run);751```752753```python754from openai import OpenAI755client = OpenAI()756757run = client.evals.runs.retrieve("YOUR_RUN_ID", eval_id="YOUR_EVAL_ID")758print(run)759```760761```ruby762require "openai"763764client = OpenAI::Client.new765run = client.evals.runs.retrieve("YOUR_RUN_ID", eval_id: "YOUR_EVAL_ID")766puts(run.id)767```768769```bash770curl https://api.openai.com/v1/evals/YOUR_EVAL_ID/runs/YOUR_RUN_ID \771 -H "Authorization: Bearer $OPENAI_API_KEY" \772 -H "Content-Type: application/json"773```774775776You'll need the UUID of both your eval and eval run to fetch its status. When you do, you'll see eval run data that looks like this:777778779```json780{781 "object": "eval.run",782 "id": "evalrun_67e44c73eb6481909f79a457749222c7",783 "eval_id": "eval_67e44c5becec81909704be0318146157",784 "report_url": "https://platform.openai.com/evaluation/evals/xxx",785 "status": "completed",786 "model": "gpt-4.1",787 "name": "Categorization text run",788 "created_at": 1743015028,789 "result_counts": {790 "total": 3,791 "errored": 0,792 "failed": 0,793 "passed": 3794 },795 "per_model_usage": [796 {797 "model_name": "gpt-4o-2024-08-06",798 "invocation_count": 3,799 "prompt_tokens": 166,800 "completion_tokens": 6,801 "total_tokens": 172,802 "cached_tokens": 0803 }804 ],805 "per_testing_criteria_results": [806 {807 "testing_criteria": "Match output to human label-40d67441-5000-4754-ab8c-181c125803ce",808 "passed": 3,809 "failed": 0810 }811 ],812 "data_source": {813 "type": "responses",814 "source": {815 "type": "file_id",816 "id": "file-J7MoX9ToHXp2TutMEeYnwj"817 },818 "input_messages": {819 "type": "template",820 "template": [821 {822 "type": "message",823 "role": "developer",824 "content": {825 "type": "input_text",826 "text": "You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of Hardware, Software, or Other. Respond with only one of those words."827 }828 },829 {830 "type": "message",831 "role": "user",832 "content": {833 "type": "input_text",834 "text": "{{item.ticket_text}}"835 }836 }837 ]838 },839 "model": "gpt-4.1",840 "sampling_params": null841 },842 "error": null,843 "metadata": {}844}845```846847848849850The API response contains granular information about test criteria results, API usage for generating model responses, and a `report_url` property that takes you to a page in the dashboard where you can explore the results visually.851852In our simple test, the model reliably generated the content we wanted for a small test case sample. In reality, you will often have to run your eval with more criteria, different prompts, and different data sets. But the process above gives you all the tools you need to build robust evals for your LLM apps!853854## Next steps855856Now you know how to create and run evals via API, and using the dashboard! Here are a few other resources that may be useful to you as you continue to improve your model results.857858[Cookbook: Detecting prompt regressions859860861862 Keep tabs on the performance of your prompts as you iterate on them.](https://developers.openai.com/cookbook/examples/evaluation/use-cases/regression)863864[Cookbook: Bulk model and prompt experimentation865866867868 Compare the results of many different prompts and models at once.](https://developers.openai.com/cookbook/examples/evaluation/use-cases/bulk-experimentation)869870[Cookbook: Monitoring stored completions871872873874 Examine stored completions to test for prompt regressions.](https://developers.openai.com/cookbook/examples/evaluation/use-cases/completion-monitoring)875876[Fine-tuning877878879880 Improve a model's ability to generate responses tailored to your use case.](https://developers.openai.com/api/docs/guides/model-optimization)881[Model distillation882883884885 Learn how to distill large model results to smaller, cheaper, and faster886 models.](https://developers.openai.com/api/docs/guides/supervised-fine-tuning#distilling-from-a-larger-model)