Repository Public
1# Reinforcement fine-tuning use cases23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45[Reinforcement fine-tuning](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning) (RFT) provides a way to improve your model's performance at specific tasks. The task must be clear and have verifiable answers.67OpenAI is winding down the fine-tuning platform. The platform is no longer8 accessible to new users, but existing users of the fine-tuning platform will9 be able to create training jobs for the coming months.10 1112 All fine-tuned models will remain available for inference until their base13 models are [deprecated](https://developers.openai.com/api/docs/deprecations). The full timeline is14 [here](https://developers.openai.com/api/docs/deprecations).1516## When to use reinforcement fine-tuning1718Agentic workflows are designed to make decisions that are both correct and verifiable. RFT can help by providing explicit rubrics and using code‑based or LLM‑based graders to measure functional success, factual accuracy, or policy compliance.1920Across early users, three clear use cases have emerged:21221. **Turn instructions into working code**: Convert open-ended prompts into structured code, configs, or templates that must pass deterministic tests.231. **Pull facts into a clean format**: Extract verifiable facts and summaries from messy, unstructured text and return JSON-structured or other schema-based outputs.241. **Apply complex rules correctly**: Make fine-grained label or policy decisions when the information provided is nuanced, large in quantity, hierarchical, or high-stakes.2526[Ready to use reinforcement fine-tuning? Skip to the guide →](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning)2728### 1. Turn instructions into working code2930In this use case, models reason over hidden domain constraints to produce structured outputs like code, queries, or infrastructure templates. Outputs must satisfy multiple correctness conditions, and success is usually deterministically graded: the artifact either compiles, passes tests, or meets an explicit schema.3132#### Wiring verification IPs for semiconductor design33343536Use case3738 3940> **Company**: [ChipStack](https://www.chipstack.ai) is building the next-generation of AI-powered tools for chip design and verification, aimed at significantly reducing the time and cost of developing and validating complex semiconductor chips.41>42> **Problem to solve**: One task that's challenging and time-consuming for humans is binding design interfaces to verification IPs (pre-created verification components that, when properly applied, can significantly enhance quality and coverage of verification). There are many verification IPs, and each can contain dozens to hundreds of signals that may be mapped. Someone must understand this domain well in order to apply the verification IP correctly.43>44> **Objective**: To train OpenAI reasoning models to do this instead, ChipStack prepared a dataset consisting of less than 50 samples, then performed several RFT variations. For the final evaluation report, they ran this evaluation set three times against each model and variation—o1-mini base and fine-tuned, o3-mini base and fine-tuned—and averaged the results per-sample then overall.454647 4849 5051 52Prompt5354 5556> Below is a piece of example data provided.5758```59[60 {“name”: “BLOCK_SIZE”, “value”: “8”},61 {“name”: “ADDR_WIDTH”, “value”: “4”}62]63```646566 6768 6970 71Grader code7273 7475> Below is a grader definition in Python of a string map, represented as a list of objects with `name` and `value` properties.76>77> Conceptually, this is meant to model a type like `Dict[str, str]`.7879```python80{81 "type": "python",82 "name": "donors_caas",83 "image_tag": "alpha",84 "source": """from collections import Counter8586def grade(sample: dict[str, str], item: dict[str, str]) -> float:87 # multisets of (name, value) pairs88 predicted = sample["output_json"]["predicted"]89 expected = item["reference_answer"]90 pred_counts = Counter((d["name"], d["value"]) for d in predicted)91 exp_counts = Counter((d["name"], d["value"]) for d in expected)9293 true_pos = sum(min(pred_counts[p], exp_counts[p]) for p in pred_counts)94 pred_total = sum(pred_counts.values())95 exp_total = sum(exp_counts.values())9697 precision = true_pos / pred_total if pred_total else 0.098 recall = true_pos / exp_total if exp_total else 0.099100 if precision + recall == 0.0:101 return 0.0102 return 2 * precision * recall / (precision + recall)""",103}104```105106107 108109 110111 112Results113114 115116> For both o1-mini and o3-mini, performance improved by ~12 percentage points. The fine-tuned variants got much better about recognizing when not to apply wiring. Many commercial verification IPs can contain hundreds of optional signals, most of which are not meant to be applied.117>118> "Thanks to powerful base models and easy-to-use Reinforced Fine-Tuning APIs, we were able to significantly boost performance on our task with a small set of high-quality samples."119>120> —[ChipStack](https://www.chipstack.ai), next-generation of AI-powered tools for chip design and verification121122123124#### Production-ready API snippets that compile and pass AST checks125126127128Use case129130 131132> **Company**: [Runloop](https://www.runloop.ai) is a platform for AI-powered coding agents to be deployed into production and built with public and custom benchmarking capabilities to refine performance.133>134> **Problem to solve**: Runloop wanted to improve model performance at using third-party APIs, such as the Stripe API, which can be large and complex without a human in the loop. If they could train a model to use the Stripe API, Runloop could turn economically impactful business cases into working code.135>136> **Objective**: Their goal was teaching the model to master usage of the Stripe API, including writing complete code snippets for arbitrary user requests by either adapting information from existing integration guides, merging information from multiple guides, or inferring information not explicitly stated in the guides. They used RFT with two primary rewards:137>138> 1. Reward the model for outputting the answer in a Markdown format that aligns with expectation of how a "dynamic" integration guide should look.139> 1. Reward the model for producing "correct" code snippets by validating the outputted code via AST Grep. This allows them to confirm the model is making the correct Stripe SDK calls with the correct parameters and in some cases even in the correct order.140141142 143144 145146 147Grader code148149 150151````python152# Note this file gets uploaded to the OpenAI API as a grader153from ast_grep_py import SgRoot154from pydantic import BaseModel, Field # type: ignore155from typing import Any156import re157158SUPPORTED_LANGUAGES = ['typescript', 'javascript', 'ts', 'js']159160class CodeBlock(BaseModel):161 language: str = Field(162 description="Programming language of the code block (e.g., 'python', 'javascript')",163 examples=["python", "javascript", "typescript"]164 )165 path: str = Field(166 description="Target file path where the code should be written",167 examples=["main.py", "src/app.js", "index.html"]168 )169 code: str = Field(170 description="Actual code content extracted from the code block"171 )172173class ASTGrepPattern(BaseModel):174 file_path_mask: str = Field(..., description="The file path pattern to match against")175 pattern: str = Field(..., description="The main AST grep pattern to search for")176 additional_greps: list[str] | None = Field(177 default=None,178 description="Additional patterns that must also be present in the matched code"179 )180181def extract_code_blocks(llm_output: str) -> list[CodeBlock]:182 # Regular expression to match code blocks with optional language and path183 try:184 pattern = r"```(\w+\s+)?([\w./-]+)?\n([\s\S]*?)\n```"185 matches = list(re.finditer(pattern, llm_output, re.DOTALL))186187 print(f"Found {len(matches)} code blocks in the LLM output")188189 # Check if any code blocks were found190 if not matches:191 raise Exception("No code blocks found in the LLM response")192193 code_blocks: list[CodeBlock] = []194 for match in matches:195 language = match.group(1) or ""196 path = match.group(2) or ""197 code = match.group(3)198199 # Clean the path and language200 path = path.strip()201 language = language.strip()202203 # If path is relative (doesn't start with /), prefix with /home/user/testbed/204 if path and not path.startswith("/"):205 original_path = path206 path = f"/home/user/testbed/{path}"207 print(208 f"Converting relative path '{original_path}' to absolute path '{path}'"209 )210211 code_blocks.append(212 CodeBlock(language=language, path=path, code=code.strip())213 )214215 # Check for missing language or path in code blocks216 missing_language = [217 i for i, block in enumerate(code_blocks) if not block.language218 ]219 missing_path = [i for i, block in enumerate(code_blocks) if not block.path]220221 if missing_language:222 print(223 f"WARNING: Code blocks at positions {missing_language} are missing language identifiers"224 )225 raise Exception(226 f"Code blocks at positions {missing_language} are missing language identifiers"227 )228229 if missing_path:230 print(231 f"WARNING: Code blocks at positions {missing_path} are missing file paths"232 )233 raise Exception(234 f"Code blocks at positions {missing_path} are missing file paths"235 )236237 paths = [block.path for block in code_blocks if block.path]238 print(239 f"Successfully extracted {len(code_blocks)} code blocks with paths: {', '.join(paths)}"240 )241242 except Exception as e:243 print(f"Error extracting code blocks: {str(e)}")244 raise245246 return code_blocks247248249def calculate_ast_grep_score(code_blocks: list[CodeBlock], ast_greps: Any) -> float:250 # Convert ast_greps to list if it's a dict251 if isinstance(ast_greps, dict):252 ast_greps = [ast_greps]253254 # Parse each grep pattern into the Pydantic model255 parsed_patterns: list[ASTGrepPattern] = []256 for grep in ast_greps:257 try:258 pattern = ASTGrepPattern(**grep)259 parsed_patterns.append(pattern)260 except Exception as e:261 print(f"Error parsing AST grep pattern: {e}")262 return 0.0263264 if not parsed_patterns:265 return 0.0266267 total_score = 0.0268 pattern_count = len(parsed_patterns)269270 # Filter code blocks to only include TypeScript and JavaScript files271 supported_blocks = [272 block for block in code_blocks273 if block.language.lower() in SUPPORTED_LANGUAGES274 ]275276 if not supported_blocks:277 print("No TypeScript or JavaScript code blocks found to analyze")278 return 0.0279280 for pattern in parsed_patterns:281 # Find matching code blocks based on path prefix282 matching_blocks = [283 block for block in supported_blocks284 if block.path.startswith(pattern.file_path_mask)285 ]286287 if not matching_blocks:288 print(f"No matching code blocks found for path prefix: {pattern.file_path_mask}")289 continue290291 pattern_found = False292 for block in matching_blocks:293 try:294 # Create AST root for the code block295 root = SgRoot(block.code, block.language)296 node = root.root()297298 # Check main pattern299 matches = node.find(pattern=pattern.pattern)300 if not matches:301 continue302303 # If we have additional greps, check them too304 if pattern.additional_greps:305 all_additional_found = True306 for additional_grep in pattern.additional_greps:307 if additional_grep not in block.code:308 all_additional_found = False309 break310311 if not all_additional_found:312 continue313314 # If we get here, we found a match with all required patterns315 pattern_found = True316 break317318 except Exception as e:319 print(f"Error processing code block {block.path}: {e}")320 continue321322 if pattern_found:323 total_score += 1.0324325 # Return average score across all patterns326 return total_score / pattern_count if pattern_count > 0 else 0.0327328def grade_format(output_text: str) -> float:329 # Find <plan> and </plan> tags330 plan_start = output_text.find('<plan>')331 plan_end = output_text.find('</plan>')332333 # Find <code> and </code> tags334 code_start = output_text.find('<code>')335 code_end = output_text.find('</code>')336337 reward = 0.0338339 if plan_start == -1 or plan_end == -1 or code_start == -1 or code_end == -1:340 print(f'missing plan or code tags. format reward: {reward}')341 return reward342 reward += 0.1 # total: 0.1343344 if not (plan_start < plan_end < code_start < code_end):345 print(f'tags present but not in the correct order. format reward: {reward}')346 return reward347 reward += 0.1 # total: 0.2348349 # Check if there are any stray tags350 plan_tags = re.findall(r'</?plan>', output_text)351 code_tags = re.findall(r'</?code>', output_text)352353 if len(plan_tags) != 2 or len(code_tags) != 2:354 print(f'found stray plan or code tags. format reward: {reward}')355 return reward356 reward += 0.2 # total: 0.4357358 # Extract content after </code> tag359 after_tags = output_text[code_end + len('</code>'):].strip()360 if after_tags:361 print(f'found text after code tags. format reward: {reward}')362 return reward363 reward += 0.2 # total: 0.6364365 # Extract content inside <plan> tags366 plan_content = output_text[plan_start + len('<plan>'):plan_end].strip()367 if not plan_content:368 print(f'no plan content found. format reward: {reward}')369 return reward370 reward += 0.1 # total: 0.7371372 # Extract content inside <code> tags373 code_content = output_text[code_start + len('<code>'):code_end].strip()374 if not code_content:375 print(f'no code content found. format reward: {reward}')376 return reward377 reward += 0.1 # total: 0.8378379 # Extract content between </plan> and <code> tags380 between_tags = output_text[plan_end + len('</plan>'):code_start].strip()381 if between_tags:382 print(f'found text between plan and code tags. format reward: {reward}')383 return reward384 reward += 0.2 # total: 1.0385386 if reward == 1.0:387 print(f'global format reward: {reward}')388389 return reward390391def grade(sample: Any, item: Any) -> float:392 try:393 output_text = sample["output_text"]394395 format_reward = grade_format(output_text)396 if format_reward < 1.0:397 return format_reward398399 # Extract code content for grading400 code_start = output_text.find('<code>')401 code_end = output_text.find('</code>')402 code_to_grade: str = output_text[code_start + len('<code>'):code_end].strip()403 code_blocks: list[CodeBlock] = []404 try:405 code_blocks = extract_code_blocks(code_to_grade)406 except Exception as e:407 print(f'error extracting code blocks: {e}')408 return 0.5409410 ast_greps = item["reference_answer"]["ast_greps"]411 ast_grep_score = calculate_ast_grep_score(code_blocks, ast_greps)412413 return (format_reward + ast_grep_score) / 2.0414 except Exception as e:415 print(f"Error during grading: {str(e)}")416 return 0.0417````418419420 421422 423424 425Results426427 428429> Looking at the total reward (format and AST Grep) together, Runloop has seen improvements of on average **12%** of the RFT model compared to the base o3-mini model on the benchmark.430>431> They implement two types of tests, one providing explicit content from the integration guides (assessing reasoning and instruction following) and one without (assessing knowledge recall). Both variants saw improvement of over **8%**.432>433> “OpenAIs RFT platform gives us access to the best generalized reasoning models in the world, with the toolset to supercharge that reasoning on problem domains important to our business.”434>435> —[Runloop](https://www.runloop.ai/)436437438439#### Correct handling of conflicts and dupes in a schedule manager440441442443Use case444445 446447> **Company**: [Milo](https://www.joinmilo.com) helps busy parents manage chaotic family schedules by converting messy inputs—like text convos with to-dos, school newsletter PDFs, weekly reminders, sports schedule emails—into reliable calendar and list actions.448>449> **Problem to solve**: Base GPT-4o prompting and SFT fell short of trust thresholds.450>451> **Objective**: Milo used RFT to properly create coding tasks like event vs. list classification, recurrence rule generation, accurate updates and deletes, conflict detection, and strict output formatting. They defined a grader that checked whether generated item objects were complete, categorized correctly, and were a duplicate or had a calendar conflict.452453454 455456 457458 459Results460461 462463> Results showed performance improvements across the board, with average correctness scores **increasing from 0.86 to 0.91**, while the most challenging scenarios improved from **0.46 to 0.71** (where a perfect score=1).464>465> "Accuracy isn't just a metric—it's peace of mind for busy parents. These are still early days but with such important improvements in base performance, we're able to push more aggressively into complex reasoning needs."466>467> "Navigating and supporting family dynamics involves understanding nuanced implications of the data. Take conflicts—knowing soccer for Ethan conflicts with Ella's recital because Dad has to drive both kids goes deeper than simple overlapping times."468>469> —[Milo](https://www.joinmilo.com), AI scheduling tool for families470471472473### 2. Pull facts into a clean format474475These tasks typically involve subtle distinctions that demand clear classification guidelines. Successful framing requires explicit and hierarchical labeling schemes defined through consensus by domain experts. Without consistent agreement, grading signals become noisy, weakening RFT effectiveness.476477#### Assigning ICD-10 medical codes478479480481Use case482483 484485> **Company**: [Ambience](https://www.ambiencehealthcare.com) is an AI platform that eliminates administrative burden for clinicians and ensures accurate, compliant documentation across 100+ specialties, helping physicians focus on patient care while increasing documentation quality and reducing compliance risk for health systems.486>487> **Problem to solve**: ICD-10 coding is one of the most intricate administrative tasks in medicine. After every patient encounter, clinicians must map each diagnosis to one of ~70,000 codes—navigating payor-specific rules on specificity, site-of-care, and mutually exclusive pairings. Errors can trigger audits and fines that stretch into nine figures.488>489> **Objective**: Using reinforcement fine-tuning on OpenAI frontier models, Ambience wanted to train a reasoning system that listens to the visit audio, pulls in relevant EHR context, and recommends ICD-10 codes with accuracy exceeding expert clinicians.490491492 493494 495496 497Results498499 500501> Ambience achieved model improvements that can lead human experts.502>503> On a gold-panel test set spanning hundreds of encounters, reinforcement fine-tuning moved the model from trailing humans to leading them by **12 points—eliminating roughly one quarter of the coding errors trained physicians make**:504>505> - o3-mini (base): 0.39 (-6 pts)506> - Physician baseline: 0.45507> - RFT-tuned o3-mini: 0.57 (+12 pts)508>509> The result is a real-time, point-of-care coding support that can raise reimbursement integrity while reducing compliance risk.510>511> “Accurate ICD-10 selection is mission-critical for compliant documentation. RFT unlocked a new level of coding precision we hadn’t seen from any foundation model and set a new bar for automated coding.”512>513> —[Ambience Healthcare](https://www.ambiencehealthcare.com)514515516517#### Extracting excerpts to support legal claims518519520521Use case522523 524525> **Company**: [Harvey](https://www.harvey.ai) is building AI that legal teams trust—and that trust hinges on retrieving precisely the right evidence from a sprawling corpora of contracts, statutes, and case law. Legal professionals aren’t satisfied with models that merely generate plausible-sounding summaries or paraphrased answers. They demand verifiable citations—passages that can be traced directly back to source documents.526>527> **Problem to solve**: Harvey’s clients use its models to triage litigation risk, construct legal arguments, and support due diligence for legal professionals—all tasks where a single missed or misquoted sentence can flip an outcome. Models must be able to parse long, dense legal documents and extract only the portions that matter.528> In practice, these inputs are often messy and inconsistent: some claims are vague, while others hinge on rare legal doctrines buried deep in boilerplate.529>530> **Objective**: The task’s requirements are to interpret nuanced legal claims, navigate long-form documents, and select on-point support with verbatim excerpts.531532533 534535 536537 538Prompt539540 541542```543## Instructions544You will be provided with a question and a text excerpt. Identify any passages in the text that are directly relevant to answering the question.545- If there are no relevant passages, return an empty list.546- Passages must be copied **exactly** from the text. Do not paraphrase or summarize.547## Excerpt548"""{text_excerpt}"""549```550551552 553554 555556 557Grader558559 560561```python562from rapidfuzz import fuzz563564565# Similarity ratio helper566def fuzz_ratio(a: str, b: str) -> float:567 """Return a normalized similarity ratio using RapidFuzz."""568 if len(a) == 0 and len(b) == 0:569 return 1.0570 return fuzz.ratio(a, b) / 100.0571572573# Main grading entrypoint (must be named \`grade\`)574def grade(sample: dict, item: dict) -> float:575 """Compute an F1‑style score for citation extraction answers using RapidFuzz."""576 model_passages = (sample.get("output_json") or {}).get("passages", [])577 ref_passages = (item.get("reference_answer") or {}).get("passages", [])578579 # If there are no reference passages, return 0.580 if not ref_passages:581 return 0.0582583 # Recall: average best match for each reference passage.584 recall_scores = []585 for ref in ref_passages:586 best = 0.0587 for out in model_passages:588 score = fuzz_ratio(ref, out)589 if score > best:590 best = score591 recall_scores.append(best)592 recall = sum(recall_scores) / len(recall_scores)593594 # Precision: average best match for each model passage.595 if not model_passages:596 precision = 0.0597 else:598 precision_scores = []599 for out in model_passages:600 best = 0.0601 for ref in ref_passages:602 score = fuzz_ratio(ref, out)603 if score > best:604 best = score605 precision_scores.append(best)606 precision = sum(precision_scores) / len(precision_scores)607608 if precision + recall == 0:609 return 0.0610611 return 2 * precision * recall / (precision + recall)612```613614615 616617 618619 620Results621622 623624> After reinforcement fine-tuning, Harvey saw a **20% increase** in the F1 score:625>626> - Baseline F1: 0.563627> - Post-RFT F1 - 0.6765628>629> Using RFT, Harvey significantly improved legal fact-extraction performance, surpassing GPT-4o efficiency and accuracy. Early trials showed RFT **winning or tying in 93% of comparisons** against GPT-4o.630>631> “The RFT model demonstrated comparable or superior performance to GPT-4o, but with significantly faster inference, proving particularly beneficial for real-world legal use cases.632>633> —[Harvey](https://www.harvey.ai), AI for legal teams634635636637### 3. Apply complex rules correctly638639This use case involves pulling verifiable facts or entities from unstructured inputs into clearly defined schemas (e.g., JSON objects, condition codes, medical codes, legal citations, or financial metrics).640641Successful extraction tasks typically benefit from precise, continuous grading methodologies—like span-level F1 scores, fuzzy text-matching metrics, or numeric accuracy checks—to evaluate how accurately the extracted information aligns with ground truth. Define explicit success criteria and detailed rubrics. Then, the model can achieve reliable, repeatable improvements.642643#### Expert-level reasoning in tax analysis644645646647Use case648649 650651> **Company**: [Accordance](https://www.accordance.com) is building a platform for tax, audit, and CPA teams.652>653> **Problem to solve**: Taxation is a highly complex domain, requiring deep reasoning across nuanced fact patterns and intricate regulations. It's also a field that continues changing.654>655> **Objective**: Accordance wanted a high-trust system for sophisticated tax scenarios while maintaining accuracy. Unlike traditional hardcoded software, it's important that their data extraction tool adapts as the tax landscape evolves.656657658 659660 661662 663Grader code664665 666667```668[+0.05] For correctly identifying Alex (33.33%), Barbara (33.33% → 20%), Chris (33.33%), and Dana (13.33%) ownership percentages669[+0.1] For correctly calculating Barbara's annual allocation as 26.67% and Dana's as 6.67% without closing of books670[+0.15] For properly allocating Alex ($300,000), Barbara ($240,030), Chris ($300,000), and Dana ($60,030) ordinary income671[+0.1] For calculating Alex's ending stock basis as $248,333 and debt basis as $75,000672[+0.05] For calculating Barbara's remaining basis after sale as $264,421673[+0.1] For calculating AAA before distributions as $1,215,000 and ending AAA as $315,000674[+0.1] For identifying all distributions as tax-free return of capital under AAA675[+0.1] For calculating Barbara's capital gain on stock sale as $223,720 ($400,000 - $176,280)676[+0.1] For explaining that closing of books would allocate based on actual half-year results677[+0.05] For identifying the ordering rules: AAA first, then E&P ($120,000), then remaining basis678[+0.05] For noting distributions exceeding $1,215,000 would be dividends up to $120,000 E&P679[+0.05] For correctly accounting for separately stated items in basis calculations (e.g., $50,000 Section 1231 gain)680```681682683 684685 686687 688Results689690 691692> By collaborating with OpenAI and their in-house tax experts, Accordance achieved:693>694> - Almost **40% improvement** in tax analysis tasks over base models695> - Superior performance compared to all other leading models on benchmarks like TaxBench696> - The RFT-trained models demonstrated an ability to handle advanced tax scenarios with high accuracy—when evaluated by tax professionals, Accordance’s fine-tuned models showed expert-level reasoning, with the potential to save thousands of hours of manual work697>698> “We’ve achieved a 38.89% improvement in our tax analysis tasks over base models and significantly outperformed all other leading models on key tax benchmarks (including TaxBench). The RFT-trained models’ abilities to handle sophisticated tax scenarios while maintaining accuracy demonstrates the readiness of reinforcement fine-tuning—and AI more broadly—for professional applications. Most importantly, RFT provides a foundation for continuous adaptation as the tax landscape evolves, ensuring sustained value and relevance. When evaluated by tax experts, our fine-tuned models demonstrated expert-level reasoning capabilities that will save thousands of professional hours—this isn’t just an incremental improvement, it’s a paradigm shift in how tax work can be done.”699>700> —[Accordance](https://www.accordance.com/), AI tax accounting company701702703704#### Enforcement of nuanced content moderation policies705706707708Use case709710 711712> **Company**: [SafetyKit](https://www.safetykit.com) is a risk and compliance platform that helps organizations make decisions across complex content moderation workflows.713>714> **Problem to solve**: These systems must handle large volumes of content and apply intricate policy logic that requires multistep reasoning. Because of the volume of data and subtle distinctions in labelling, these types of tasks can be difficult for general purpose models.715>716> **Objective**: SafetyKit aimed to replace multiple nodes in their most complex workflows with a single reasoning agent using a reinforcement fine-tuned model. The goal is to reduce SafetyKit’s time-to-market for novel policy enforcements even in challenging, nuanced domains.717718719 720721 722723 724Results725726 727728> SafetyKit is using their o3-mini RFT model to support advanced content moderation capabilities, ensuring user safety for one of the largest AI chatbot companies in the world. They have successfully improved F1-score **from 86% to 90%**, soon to replace dozens of 4o calls within their production pipeline.729>730> "SafetyKit’s RFT-enabled moderation achieved substantial improvements in nuanced content moderation tasks, crucial for safeguarding users in dynamic, real-world scenarios."731>732> —[SafetyKit](https://www.safetykit.com)733734735736#### Legal document reviews, comparisons, and summaries737738739740Use case741742 743744> **Company**: [Thomson Reuters](https://www.thomsonreuters.com) is an AI and technology company empowering professionals with trusted content and workflow automation.745>746> **Problem to solve**: Legal professionals must read through large amounts of content before making any decisions. Thomson Reuter's CoCounsel product is designed to help these experts move faster by providing an AI assistant with content and industry knowledge. The models that power this tool must understand complex legal rules.747>748> **Objective**: Thomson Reuters aimed to create a reinforcement fine-tuned model excelling in legal AI skills. They conducted preliminary evaluations of RFT to see if they could achieve model performance improvements, using specialized datasets from three highly-used CoCounsel Legal AI skills for legal professionals:749>750> 1. Review documents: Generates detailed answers to questions asked against contracts, transcripts, and other legal documents751> 1. Compare documents: Highlights substantive differences between two or more different contracts or documents752> 1. Summarize: Summarizes the most important information within one or more documents to enable rapid legal review753754755 756757 758759 760Results761762 763764> 765>766> "LLM as a judge has been helpful in demonstrating the possibility of improving upon the reasoning models - in preliminary evaluations, the RFT model consistently performed better than the baseline o3-mini and o1 model"767>768> —[Thomson Reuters](https://www.thomsonreuters.com/), AI and technology company769770771772## Evals are the foundation773774**Before implementing RFT, we strongly recommended creating and running an eval for the task you intend to fine-tune on**. If the model you intend to fine-tune scores at either the absolute minimum or absolute maximum possible score, then RFT won’t be useful to you.775776RFT works by reinforcing better answers to provided prompts. If we can’t distinguish the quality of different answers (i.e., if they all receive the minimum or maximum possible score), then there's no training signal to learn from. However, if your eval scores somewhere in the range between the minimum and maximum possible scores, there's enough data to work with.777778An effective eval reveals opportunities where human experts consistently agree but current frontier models struggle, presenting a valuable gap for RFT to close. [Get started with evals](https://developers.openai.com/api/docs/guides/evals).779780## How to get better results from RFT781782To see improvements in your fine-tuned model, there are two main places to revisit and refine: making sure your task is well defined, and making your grading scheme more robust.783784### Reframe or clarify your task785786Good tasks give the model a fair chance to learn and let you quantify improvements.787788- **Start with a task the model can already solve occasionally**. RFT works by sampling many answers, keeping what looks best, and nudging the model toward those answers. If the model never gets the answer correct today, it cannot improve.789- **Make sure each answer can be graded**. A grader must read an answer and produce a score without a person in the loop. We support multiple [grader types](https://developers.openai.com/api/docs/guides/graders), including custom Python graders and LLM judges. If you can't write code to judge the answer with an available grader, RFT is not the right tool.790- **Remove doubt about the “right” answer**. If two careful people often disagree on the solution, the task is too fuzzy. Rewrite the prompt, add context, or split the task into clearer parts until domain experts agree.791- **Limit lucky guesses**. If the task is multiple choice with one obvious best pick, the model can win by chance. Add more classes, ask for short open‑ended text, or tweak the format so guessing is costly.792793### Strengthen your grader794795Clear, robust grading schemes are essential for RFT.796797- **Produce a smooth score, not a pass/fail stamp**. A score that shifts gradually as answers improve provides a better training signal.798- **Guard against reward hacking**. This happens when the model finds a shortcut that earns high scores without real skill.799- **Avoid skewed data**. Datasets in which one label shows up most of the time invite the model to guess that label. Balance the set or up‑weight rare cases so the model must think.800- **Use an LLM judge when code falls short**. For rich, open‑ended answers, have a [separate OpenAI model grade](https://developers.openai.com/api/docs/guides/graders#model-graders) your fine-tuned model's answers. Make sure you:801 - **Evaluate the judge**: Run multiple candidate responses and correct answers through your LLM judge to ensure the grade returned is stable and aligned with preference.802 - **Provide few-shot examples**. Include great, fair, and poor answers in the prompt to improve the grader's effectiveness.803804Learn more about [grader types](https://developers.openai.com/api/docs/guides/graders).805806## Other resources807808For more inspiration, visit the [OpenAI Cookbook](https://developers.openai.com/cookbook), which contains example code and links to third-party resources, or learn more about our models and reasoning capabilities:809810- [Meet the models](https://developers.openai.com/api/docs/models)811- [Reinforcement fine-tuning guide](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning)812- [Graders](https://developers.openai.com/api/docs/guides/graders)813- [Model optimization overview](https://developers.openai.com/api/docs/guides/model-optimization)