Repository Public
1# Direct preference optimization23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45[Direct Preference Optimization](https://arxiv.org/abs/2305.18290) (DPO) fine-tuning allows you to fine-tune models based on prompts and pairs of responses. This approach enables the model to learn from more subjective human preferences, optimizing for outputs that are more likely to be favored. DPO is currently only supported for text inputs and outputs.67OpenAI is winding down the fine-tuning platform. The platform is no longer8 accessible to new users, but existing users of the fine-tuning platform will9 be able to create training jobs for the coming months.10 1112 All fine-tuned models will remain available for inference until their base13 models are [deprecated](https://developers.openai.com/api/docs/deprecations). The full timeline is14 [here](https://developers.openai.com/api/docs/deprecations).1516171819<table>20<tbody>21<tr>22<th>How it works</th>23<th>Best for</th>24<th>Use with</th>25</tr>2627<tr>28<td>29Provide both a correct and incorrect example response for a prompt. Indicate the correct response to help the model perform better.30</td>31<td>32- Summarizing text, focusing on the right things33- Generating chat messages with the right tone and style34</td>35<td>36`gpt-4.1-2025-04-14`37`gpt-4.1-mini-2025-04-14`38`gpt-4.1-nano-2025-04-14`39</td>40</tr>41</tbody>42</table>4344## Data format4546Each example in your dataset should contain:4748- A prompt, like a user message.49- A preferred output (an ideal assistant response).50- A non-preferred output (a suboptimal assistant response).5152The data should be formatted in JSONL format, with each line [representing an example](https://developers.openai.com/api/reference/resources/fine_tuning) in the following structure:5354```json55{56 "input": {57 "messages": [58 {59 "role": "user",60 "content": "Hello, can you tell me how cold San Francisco is today?"61 }62 ],63 "tools": [],64 "parallel_tool_calls": true65 },66 "preferred_output": [67 {68 "role": "assistant",69 "content": "Today in San Francisco, it is not quite cold as expected. Morning clouds will give away to sunshine, with a high near 68°F (20°C) and a low around 57°F (14°C)."70 }71 ],72 "non_preferred_output": [73 {74 "role": "assistant",75 "content": "It is not particularly cold in San Francisco today."76 }77 ]78}79```8081Currently, we only train on one-turn conversations for each example, where the preferred and non-preferred messages need to be the last assistant message.8283## Create a DPO fine-tune job8485Uploading training data and using a model fine-tuned with DPO follows the [same flow described here](https://developers.openai.com/api/docs/guides/model-optimization).8687To create a DPO fine-tune job, use the `method` field in the [fine-tuning job creation endpoint](https://developers.openai.com/api/reference/resources/fine_tuning), where you can specify `type` as well as any associated `hyperparameters`. For DPO:8889- set the `type` parameter to `dpo`90- optionally set the `hyperparameters` property with any options you'd like to configure.9192The `beta` hyperparameter is a new option that is only available for DPO. It's a floating point number between `0` and `2` that controls how strictly the new model will adhere to its previous behavior, versus aligning with the provided preferences. A high number will be more conservative (favoring previous behavior), and a lower number will be more aggressive (favor the newly provided preferences more often).9394You can also set this value to `auto` (the default) to use a value configured by the platform.9596The example below shows how to configure a DPO fine-tuning job using the OpenAI SDK.9798Create a fine-tuning job with DPO99100```javascript101import OpenAI from "openai";102103const openai = new OpenAI();104105const job = await openai.fineTuning.jobs.create({106 training_file: "file-all-about-the-weather",107 model: "gpt-4o-2024-08-06",108 method: {109 type: "dpo",110 dpo: {111 hyperparameters: { beta: 0.1 },112 },113 },114});115```116117```python118from openai import OpenAI119120client = OpenAI()121122job = client.fine_tuning.jobs.create(123 training_file="file-all-about-the-weather",124 model="gpt-4o-2024-08-06",125 method={126 "type": "dpo",127 "dpo": {128 "hyperparameters": {"beta": 0.1},129 },130 },131)132```133134```go135package main136137import (138 "context"139 "fmt"140141 "github.com/openai/openai-go/v3"142)143144func main() {145 client := openai.NewClient()146 job, err := client.FineTuning.Jobs.New(context.Background(), openai.FineTuningJobNewParams{147 TrainingFile: "file-all-about-the-weather",148 Model: "gpt-4o-2024-08-06",149 Method: openai.FineTuningJobNewParamsMethod{150 Type: "dpo",151 Dpo: openai.DpoMethodParam{Hyperparameters: openai.DpoHyperparameters{152 Beta: openai.DpoHyperparametersBetaUnion{OfFloat: openai.Float(0.1)},153 }},154 },155 })156 if err != nil {157 panic(err)158 }159 fmt.Println(job.ID)160}161```162163```java164import com.openai.client.OpenAIClient;165import com.openai.client.okhttp.OpenAIOkHttpClient;166import com.openai.models.finetuning.jobs.JobCreateParams;167import com.openai.models.finetuning.methods.DpoHyperparameters;168import com.openai.models.finetuning.methods.DpoMethod;169170String fileId = "file-all-about-the-weather";171172var job =173 client174 .fineTuning()175 .jobs()176 .create(177 JobCreateParams.builder()178 .model("gpt-4.1-mini-2025-04-14")179 .trainingFile(fileId)180 .method(181 JobCreateParams.Method.builder()182 .type(JobCreateParams.Method.Type.DPO)183 .dpo(184 DpoMethod.builder()185 .hyperparameters(DpoHyperparameters.builder().beta(0.1).build())186 .build())187 .build())188 .build());189190System.out.println(job.id());191```192193```ruby194require "openai"195196client = OpenAI::Client.new197job = client.fine_tuning.jobs.create(198 model: "gpt-4.1-mini-2025-04-14",199 training_file: "file-all-about-the-weather",200 method_: {201 type: :dpo,202 dpo: { hyperparameters: { beta: 0.1 } }203 }204)205puts(job.id)206```207208209## Use SFT and DPO together210211Currently, OpenAI offers [supervised fine-tuning (SFT)](https://developers.openai.com/api/docs/guides/supervised-fine-tuning) as the default method for fine-tuning jobs. Performing SFT on your preferred responses (or a subset) before running another DPO job afterwards can significantly enhance model alignment and performance. By first fine-tuning the model on the desired responses, it can better identify correct patterns, providing a strong foundation for DPO to refine behavior.212213A recommended workflow is as follows:2142151. Fine-tune the base model with SFT using a subset of your preferred responses. Focus on ensuring the data quality and representativeness of the tasks.2162. Use the SFT fine-tuned model as the starting point, and apply DPO to adjust the model based on preference comparisons.217218## Safety checks219220Before launching in production, review and follow the following safety information.221222223224### How we assess for safety225226227228Once a fine-tuning job is completed, we assess the resulting model’s behavior across 13 distinct safety categories. Each category represents a critical area where AI outputs could potentially cause harm if not properly controlled.229230| Name | Description |231| :--------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |232| advice | Advice or guidance that violates our policies. |233| harassment/threatening | Harassment content that also includes violence or serious harm towards any target. |234| hate | Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. Hateful content aimed at non-protected groups (e.g., chess players) is harassment. |235| hate/threatening | Hateful content that also includes violence or serious harm towards the targeted group based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. |236| highly-sensitive | Highly sensitive data that violates our policies. |237| illicit | Content that gives advice or instruction on how to commit illicit acts. A phrase like "how to shoplift" would fit this category. |238| propaganda | Praise or assistance for ideology that violates our policies. |239| self-harm/instructions | Content that encourages performing acts of self-harm, such as suicide, cutting, and eating disorders, or that gives instructions or advice on how to commit such acts. |240| self-harm/intent | Content where the speaker expresses that they are engaging or intend to engage in acts of self-harm, such as suicide, cutting, and eating disorders. |241| sensitive | Sensitive data that violates our policies. |242| sexual/minors | Sexual content that includes an individual who is under 18 years old. |243| sexual | Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness). |244| violence | Content that depicts death, violence, or physical injury. |245246Each category has a predefined pass threshold; if too many evaluated examples in a given category fail, OpenAI blocks the fine-tuned model from deployment. If your fine-tuned model does not pass the safety checks, OpenAI sends a message in the fine-tuning job explaining which categories don't meet the required thresholds. You can view the results in the moderation checks section of the fine-tuning job.247248249250251252253254### How to pass safety checks255256257258In addition to reviewing any failed safety checks in the fine-tuning job object, you can retrieve details about which categories failed by querying the [fine-tuning API events endpoint](https://platform.openai.com/docs/api-reference/fine-tuning/list-events). Look for events of type `moderation_checks` for details about category results and enforcement. This information can help you narrow down which categories to target for retraining and improvement. The [model spec](https://cdn.openai.com/spec/model-spec-2024-05-08.html#overview) has rules and examples that can help identify areas for additional training data.259260While these evaluations cover a broad range of safety categories, conduct your own evaluations of the fine-tuned model to ensure it's appropriate for your use case.261262263264265266## Next steps267268Now that you know the basics of DPO, explore these other methods as well.269270[Supervised fine-tuning271272273274 Fine-tune a model by providing correct outputs for sample inputs.](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)275276[Vision fine-tuning277278279280 Learn to fine-tune for computer vision with image inputs.](https://developers.openai.com/api/docs/guides/vision-fine-tuning)281282[Reinforcement fine-tuning283284285286 Fine-tune a reasoning model by grading its outputs.](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning)