Repository Public
1# Graders23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Graders are a way to evaluate your model's performance against reference answers. Our [graders API](https://developers.openai.com/api/reference/resources/graders) is a way to test your graders, experiment with results, and improve your fine-tuning or evaluation framework to get the results you want.67OpenAI is deprecating graders as part of the evals and fine-tuning workflows8 they support. See the [deprecations page](https://developers.openai.com/api/docs/deprecations) for the9 current transition timelines.1011## Overview1213Graders let you compare reference answers to the corresponding model-generated answer and return a grade in the range from 0 to 1. It's sometimes helpful to give the model partial credit for an answer, rather than a binary 0 or 1.1415Graders are specified in JSON format, and there are several types:1617- [String check](#string-check-graders)18- [Text similarity](#text-similarity-graders)19- [Score model grader](#score-model-graders)20- [Python code execution](#python-graders)2122In reinforcement fine-tuning, you can nest and combine graders by using [`multigrader` objects](#combined-graders).2324Use this guide to learn about each grader type and see starter examples. To build a grader and get started with reinforcement fine-tuning, see the [RFT guide](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning). Or to get started with evals, see the [Evals guide](https://developers.openai.com/api/docs/guides/evals).2526## Templating2728The inputs to certain graders use a templating syntax to grade multiple examples with the same configuration. Any string with `{{ }}` double curly braces will be substituted with the variable value.2930Each input inside the `{{}}` must include a _namespace_ and a _variable_ with the following format `{{ namespace.variable }}`. The only supported namespace values are `item` and `sample`.3132All nested variables can be accessed with JSON path like syntax.3334### Item namespace3536The item namespace will be populated with variables from the input data source for evals, and from each dataset item for fine-tuning. For example, if a row contains the following3738```json39{40 "reference_answer": "..."41}42```4344This can be used within the grader as `{{ item.reference_answer }}`.4546### Sample namespace4748The sample namespace will be populated with variables from the model sampling step during evals or during the fine-tuning step. The following variables are included4950- `output_text`, the model output content as a string.51- `output_json`, the model output content as a JSON object, only if `response_format` is included in the sample.52- `output_tools`, the model output `tool_calls`, which have the same structure as output tool calls in the [chat completions API](https://developers.openai.com/api/reference/resources/chat).53- `choices`, the output choices, which has the same structure as output choices in the [chat completions API](https://developers.openai.com/api/reference/resources/chat).54- `output_audio`, the model audio output object containing Base64-encoded `data` and a `transcript`.5556For example, to access the model output content as a string, `{{ sample.output_text }}` can be used within the grader.57585960#### Details on grading tool calls61626364When training a model to improve tool-calling behavior, you will need to write your grader to operate over the `sample.output_tools` variable. The contents of this variable will be the same as the contents of the `response.choices[0].message.tool_calls` ([see function calling docs](https://developers.openai.com/api/docs/guides/function-calling?api-mode=chat)).6566A common way of grading tool calls is to use two graders, one that checks the name of the tool that is called and another that checks the arguments of the called function. An example of a grader that does this is shown below:6768```json69{70 "type": "multi",71 "graders": {72 "function_name": {73 "name": "function_name",74 "type": "string_check",75 "input": "get_acceptors",76 "reference": "{{sample.output_tools[0].function.name}}",77 "operation": "eq"78 },79 "arguments": {80 "name": "arguments",81 "type": "string_check",82 "input": "{\"smiles\": \"{{item.smiles}}\"}",83 "reference": "{{sample.output_tools[0].function.arguments}}",84 "operation": "eq"85 }86 },87 "calculate_output": "0.5 * function_name + 0.5 * arguments"88}89```9091This is a `multi` grader that combined two simple `string_check` graders, the first checks the name of the tool called via the `sample.output_tools[0].function.name` variable, and the second checks the arguments of the called function via the `sample.output_tools[0].function.arguments` variable. The `calculate_output` field is used to combine the two scores into a single score.9293The `arguments` grader is prone to under-rewarding the model if the function arguments are subtly incorrect, like if `1` is submitted instead of the floating point `1.0`, or if a state name is given as an abbreviation instead of spelling it out. To avoid this, you can use a `text_similarity` grader instead of a `string_check` grader, or a `score_model` grader to have a LLM check for semantic similarity.949596979899## String check graders100101Use these basic string operations to return a 0 or 1. String check graders are good for scoring straightforward pass or fail answers—for example, the correct name of a city, a yes or no answer, or an answer containing or starting with the correct information.102103```json104{105 "type": "string_check",106 "name": string,107 "operation": "eq" | "ne" | "like" | "ilike",108 "input": string,109 "reference": string,110}111```112113Operations supported for string-check-grader are:114115- `eq`: Returns 1 if the input matches the reference (case-sensitive), 0 otherwise116- `neq`: Returns 1 if the input does not match the reference (case-sensitive), 0 otherwise117- `like`: Returns 1 if the input contains the reference (case-sensitive), 0 otherwise118- `ilike`: Returns 1 if the input contains the reference (not case-sensitive), 0 otherwise119120## Text similarity graders121122Use text similarity graders when to evaluate how close the model-generated output is to the reference, scored with various evaluation frameworks.123124This is useful for open-ended text responses. For example, if your dataset contains reference answers from experts in paragraph form, it's helpful to see how close your model-generated answer is to that content, in numerical form.125126```json127{128 "type": "text_similarity",129 "name": string,130 "input": string,131 "reference": string,132 "pass_threshold": number,133 "evaluation_metric": "fuzzy_match" | "bleu" | "gleu" | "meteor" | "cosine" | "rouge_1" | "rouge_2" | "rouge_3" | "rouge_4" | "rouge_5" | "rouge_l"134}135```136137Operations supported for `string-similarity-grader` are:138139- `fuzzy_match`: Fuzzy string match between input and reference, using `rapidfuzz`140- `bleu`: Computes the BLEU score between input and reference141- `gleu`: Computes the Google BLEU score between input and reference142- `meteor`: Computes the METEOR score between input and reference143- `cosine`: Computes Cosine similarity between embedded input and reference, using `text-embedding-3-large`. Only available for evals.144- `rouge-*`: Computes the ROUGE score between input and reference145146## Model graders147148In general, using a model grader means prompting a separate model to grade the outputs of the model you're fine-tuning. Your two models work together to do reinforcement fine-tuning. The _grader model_ evaluates the _training model_.149150### Score model graders151152A score model grader will take the input and return a numeric score based on the prompt within the given range.153154```json155{156 "type": "score_model",157 "name": string,158 "input": Message[],159 "model": string,160 "pass_threshold": number,161 "range": number[],162 "sampling_params": {163 "seed": number,164 "top_p": number,165 "temperature": number,166 "max_completions_tokens": number,167 "reasoning_effort": "minimal" | "low" | "medium" | "high"168 }169}170```171172Where each message is of the following form:173174```json175{176 "role": "system" | "developer" | "user" | "assistant",177 "content": str178}179180```181182To use a score model grader, the input is a list of chat messages, each containing a `role` and `content`. The output of the grader will be truncated to the given `range`, and default to 0 for all non-numeric outputs.183Within each message, the same templating can be used as with other common graders to reference the ground truth or model sample.184185Here’s a full runnable code sample:186187```python188import os189import requests190191# get the API key from environment192api_key = os.environ["OPENAI_API_KEY"]193headers = {"Authorization": f"Bearer {api_key}"}194195# Define a score-model grader.196grader = {197 "type": "score_model",198 "name": "my_score_model",199 "input": [200 {201 "role": "system",202 "content": "You are an expert grader. If the reference and model answer are exact matches, output a score of 1. If they are somewhat similar in meaning, output a score in 0.5. Otherwise, give a score of 0.",203 },204 {205 "role": "user",206 "content": "Reference: {{ item.reference_answer }}. Model answer: {{ sample.output_text }}",207 },208 ],209 "pass_threshold": 0.5,210 "model": "o4-mini-2025-04-16",211 "range": [0, 1],212 "sampling_params": {213 "max_completions_tokens": 32768,214 "top_p": 1,215 "reasoning_effort": "medium",216 },217}218219# validate the grader220payload = {"grader": grader}221response = requests.post(222 "https://api.openai.com/v1/fine_tuning/alpha/graders/validate",223 json=payload,224 headers=headers,225)226print("validate response:", response.text)227228# run the grader with a test reference and sample229payload = {"grader": grader, "item": {"reference_answer": 1.0}, "model_sample": "0.9"}230response = requests.post(231 "https://api.openai.com/v1/fine_tuning/alpha/graders/run",232 json=payload,233 headers=headers,234)235print("run response:", response.text)236```237238```ruby239require "openai"240241client = OpenAI::Client.new242grader = {243 "type" => "score_model",244 "name" => "my_score_model",245 "input" => [246 {247 "role" => "system",248 "content" => "You are an expert grader. If the reference and model answer are exact matches, output a score of 1. If they are somewhat similar in meaning, output a score in 0.5. Otherwise, give a score of 0."249 }, {250 "role" => "user",251 "content" => "Reference: {{ item.reference_answer }}. Model answer: {{ sample.output_text }}"252 }253 ],254 "pass_threshold" => 0.5,255 "model" => "o4-mini-2025-04-16",256 "range" => [0, 1],257 "sampling_params" => {258 "max_completions_tokens" => 32768,259 "top_p" => 1,260 "reasoning_effort" => "medium"261 }262}263item = { reference_answer: 1.0 }264model_sample = "0.9"265266pp(client.fine_tuning.alpha.graders.validate(grader: grader))267pp(client.fine_tuning.alpha.graders.run(grader: grader, item: item, model_sample: model_sample))268```269270271#### Score model grader outputs272273Under the hood, the `score_model` grader will query the requested model with the provided prompt and sampling parameters and will request a response in a specific response format. The response format that is used is provided below274275```json276{277 "result": float,278 "steps": ReasoningStep[],279}280```281282Where each reasoning step is of the form283284```json285{286 description: string,287 conclusion: string288}289```290291This format queries the model not just for the numeric `result` (the reward value for the query), but also provides the model some space to think through the reasoning behind the score. When you are writing your grader prompt, it may be useful to refer to these two fields by name explicitly (for example, “include reasoning about the type of chemical bonds present in the molecule in the conclusion of your reasoning step,” or “return a value of −1.0 in the `result` field if the inputs do not satisfy condition X”).292293### Model grader constraints294295- Only the following models are supported for the `model` parameter296 - `gpt-4o-2024-08-06`297 - `gpt-4o-mini-2024-07-18`298 - `gpt-4.1-2025-04-14`299 - `gpt-4.1-mini-2025-04-14`300 - `gpt-4.1-nano-2025-04-14`301 - `o1-2024-12-17`302 - `o3-mini-2025-01-31`303 - `o3-2025-04-16`304 - `o4-mini-2025-04-16`305- `temperature` changes not supported for reasoning models.306- `reasoning_effort` is not supported for non-reasoning models.307308### How to write grader prompts309310Writing grader prompts is an iterative process. The best way to iterate on a model grader prompt is to create a model grader eval. To do this, you need:3113121. **Task prompts**: Write extremely detailed prompts for the desired task, with step-by-step instructions and many specific examples in context.3131. **Answers generated by a model or human expert**: Provide many high quality examples of answers, both from the model and trusted human experts.3141. **Corresponding ground truth grades for those answers**: Establish what a good grade looks like. For example, your human expert grades should be 1.315316Then you can automatically evaluate how effectively the model grader distinguishes answers of different quality levels. Over time, add edge cases into your model grader eval as you discover and patch them with changes to the prompt.317318For example, say you know from your human experts which answers are best:319320```321answer_1 > answer_2 > answer_3322```323324Verify that the model grader's answers match that:325326```327model_grader(answer_1, reference_answer) > model_grader(answer_2, reference_answer) > model_grader(answer_3, reference_answer)328```329330### Grader hacking331332Models being trained sometimes learn to exploit weaknesses in model graders, also known as “grader hacking” or “reward hacking." You can detect this by checking the model's performance across model grader evals and expert human evals. A model that's hacked the grader will score highly on model grader evals but score poorly on expert human evaluations. Over time, we intend to improve observability in the API to make it easier to detect this during training.333334## Python graders335336This grader allows you to execute arbitrary python code to grade the model output. The grader expects a grade function to be present that takes in two arguments and outputs a float value. Any other result (exception, invalid float value, etc.) will be marked as invalid and return a 0 grade.337338```json339{340 "type": "python",341 "source": "def grade(sample, item):\n return 1.0",342 "image_tag": "2025-05-08"343}344```345346The python source code must contain a grade function that takes in exactly two arguments and returns a float value as a grade.347348```python349from typing import Any350351352def grade(sample: dict[str, Any], item: dict[str, Any]) -> float:353 # your logic here354 return 1.0355```356357358The first argument supplied to the grading function will be a dictionary populated with the model’s output during training for you to grade. `output_json` will only be populated if the output uses `response_format`.359360```json361{362 "choices": [...],363 "output_text": "...",364 "output_json": {},365 "output_tools": [...],366 "output_audio": {}367}368```369370The second argument supplied is a dictionary populated with input grading context. For evals, this will include keys from the data source. For fine-tuning this will include keys from each training data row.371372```json373{374 "reference_answer": "...",375 "my_key": {...}376}377```378379Here's a working example. For Ruby, save the `grade` function shown above, including its import, as `grader.py`. Place `grader.py` in the directory where you run the example. The supplied function returns `1.0`; replace its body with your grading logic.380381```python382import os383import requests384385# get the API key from environment386api_key = os.environ["OPENAI_API_KEY"]387headers = {"Authorization": f"Bearer {api_key}"}388389grading_function = """390from rapidfuzz import fuzz, utils391392def grade(sample, item) -> float:393 output_text = sample["output_text"]394 reference_answer = item["reference_answer"]395 return fuzz.WRatio(output_text, reference_answer, processor=utils.default_process) / 100.0396"""397398# Define a Python grader.399grader = {"type": "python", "source": grading_function}400401# validate the grader402payload = {"grader": grader}403response = requests.post(404 "https://api.openai.com/v1/fine_tuning/alpha/graders/validate",405 json=payload,406 headers=headers,407)408print("validate request_id:", response.headers["x-request-id"])409print("validate response:", response.text)410411# run the grader with a test reference and sample412payload = {413 "grader": grader,414 "item": {"reference_answer": "fuzzy wuzzy had no hair"},415 "model_sample": "fuzzy wuzzy was a bear",416}417response = requests.post(418 "https://api.openai.com/v1/fine_tuning/alpha/graders/run",419 json=payload,420 headers=headers,421)422print("run request_id:", response.headers["x-request-id"])423print("run response:", response.text)424```425426```ruby427require "openai"428429client = OpenAI::Client.new430# Save your Python grading function as grader.py before running this example.431grading_function = File.read("grader.py")432grader = {433 type: :python,434 source: grading_function435}436item = { reference_answer: "fuzzy wuzzy had no hair" }437model_sample = "fuzzy wuzzy was a bear"438439pp(client.fine_tuning.alpha.graders.validate(grader: grader))440pp(client.fine_tuning.alpha.graders.run(grader: grader, item: item, model_sample: model_sample))441```442443444**Tip:**445If you don't want to manually put your grading function in a string, you can also load it from a Python file using `importlib` and `inspect`. For example, if your grader function is in a file named `grader.py`, you can do:446447```python448import importlib449import inspect450451grader_module = importlib.import_module("grader")452grader = {"type": "python", "source": inspect.getsource(grader_module)}453```454455456This will automatically use the entire source code of your `grader.py` file as the grader which can be helpful for longer graders.457458### Technical constraints459460- Your uploaded code must be less than `256kB` and will not have network access.461- The grading execution itself is limited to 2 minutes.462- At runtime you will be given a limit of 2Gb of memory and 1Gb of disk space to use.463- There's a limit of 2 CPU cores—any usage above this amount will result in throttling464465The following third-party packages are available at execution time for the image tag `2025-05-08`466467```468numpy==2.2.4469scipy==1.15.2470sympy==1.13.3471pandas==2.2.3472rapidfuzz==3.10.1473scikit-learn==1.6.1474rouge-score==0.1.2475deepdiff==8.4.2476jsonschema==4.23.0477pydantic==2.10.6478pyyaml==6.0.2479nltk==3.9.1480sqlparse==0.5.3481rdkit==2024.9.6482scikit-bio==0.6.3483ast-grep-py==0.36.2484```485486Additionally, the following NLTK corpora are available:487488```489punkt490stopwords491wordnet492omw-1.4493names494```495496## Combined graders497498> Currently, this grader is only used for Reinforcement fine-tuning499500A `multigrader` object combines the output of multiple graders to produce a single score. Combined graders compute grades over the fields of other grader objects and turn those sub-grades into an overall grade. This is useful when a correct answer depends on multiple things being true—for example, that the text is similar _and_ that the answer contains a specific string.501502As an example, say you wanted the model to output JSON with the following two fields:503504```json505{506 "name": "John Doe",507 "email": "john.doe@gmail.com"508}509```510511You'd want your grader to compare the two fields and then take the average between them.512513You can do this by combining multiple graders into an object grader, and then defining a formula to calculate the output score based on each field:514515```json516{517 "type": "multi",518 "graders": {519 "name": {520 "name": "name_grader",521 "type": "text_similarity",522 "input": "{{sample.output_json.name}}",523 "reference": "{{item.name}}",524 "evaluation_metric": "fuzzy_match",525 "pass_threshold": 0.9526 },527 "email": {528 "name": "email_grader",529 "type": "string_check",530 "input": "{{sample.output_json.email}}",531 "reference": "{{item.email}}",532 "operation": "eq"533 }534 },535 "calculate_output": "(name + email) / 2"536}537```538539In this example, it’s important for the model to get the email exactly right (`string_check` returns either 0 or 1) but we tolerate some misspellings on the name (`text_similarity` returns range from 0 to 1). Samples that get the email wrong will score between 0-0.5, and samples that get the email right will score between 0.5-1.0.540541You cannot nest one `multigrader` inside another.542543The calculate output field will have the keys of the input `graders` as possible variables and the following features are supported:544545**Operators**546547- `+` (addition)548- `-` (subtraction)549- `*` (multiplication)550- `/` (division)551- `^` (power)552553**Functions**554555- `min`556- `max`557- `abs`558- `floor`559- `ceil`560- `exp`561- `sqrt`562- `log`563564## Limitations and tips565566Designing and creating graders is an iterative process. Start small, experiment, and continue to make changes to get better results.567568### Design tips569570To get the most value from your graders, use these design principles:571572- **Produce a smooth score, not a pass/fail stamp**. A score that shifts gradually as answers improve helps the optimizer see which changes matter.573- **Guard against reward hacking**. This happens when the model finds a shortcut that earns high scores without real skill. Make it hard to loophole your grading system.574- **Avoid skewed data**. Datasets in which one label shows up most of the time invite the model to guess that label. Balance the set or up‑weight rare cases so the model must think.575- **Use an LLM‑as‑a-judge when code falls short**. For rich, open‑ended answers, ask another language model to grade. When building LLM graders, run multiple candidate responses and ground truths through your LLM judge to ensure grading is stable and aligned with preference. Provide few-shot examples of great, fair, and poor answers in the prompt.