Repository Public
1# Moderation23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Use OpenAI moderation models to detect harmful content in text and images. You can classify standalone inputs with the [moderation endpoint](https://developers.openai.com/api/reference/resources/moderations) or request moderation scores alongside a generated response. Use the results to enforce your application's policy, such as filtering content, routing a request for review, or intervening with accounts that submit flagged content.67The `omni-moderation-latest` model accepts text and image inputs. It doesn't classify audio. The moderation endpoint is free to use, and image files can be up to 20 MB.89**Child safety:** Do not send known or suspected child sexual abuse material (CSAM) to the Moderation API. The API is not designed for CSAM detection or handling and is not a substitute for dedicated child-safety safeguards. See our [CSAM guidance](https://developers.openai.com/api/docs/guides/csam-guidance) for steps to prevent, detect, respond to, and report CSAM.1011## Choose a moderation workflow1213| Workflow | Use when |14| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |15| [Moderate generated content](#moderate-generated-content) | Your application generates text with the Responses API or Chat Completions API and needs moderation signals. |16| [Classify standalone inputs](#classify-standalone-inputs) | Your application needs to classify text or images without generating a model response. |17| [Understand moderation results](#understand-moderation-results) | Your application needs to interpret flags, categories, scores, or applied input types. |18| [Review supported categories](#review-supported-categories) | Your application needs to know which harm categories apply to text, images, or both. |1920## Moderate generated content2122When your application needs generated text and moderation scores together, pass a top-level `moderation` object in the generation request. The API returns moderation scores for the model input and generated output without a separate moderation request.2324The model still generates normally. Review the moderation results before you show the output to a user or take downstream actions.25262728Set `moderation.model` when you create a response:2930Generate a response with moderation scores3132```javascript33import OpenAI from "openai";3435const client = new OpenAI();3637const response = await client.responses.create({38 model: "gpt-6-astra",39 input: [40 {41 role: "user",42 content:43 "A user asks for instructions to make a harmful weapon. Draft a brief refusal and offer a safer alternative.",44 },45 ],46 moderation: { model: "omni-moderation-latest" },47});4849const inputModeration = response.moderation.input;50const outputModeration = response.moderation.output;51if (inputModeration.type === "error") {52 throw new Error(inputModeration.message);53}54if (outputModeration.type === "error") {55 throw new Error(outputModeration.message);56}5758console.log(inputModeration.flagged);59console.log(outputModeration.flagged);60```6162```python63from openai import OpenAI6465client = OpenAI()6667response = client.responses.create(68 model="gpt-6-astra",69 input=[70 {71 "role": "user",72 "content": (73 "A user asks for instructions to make a harmful weapon. "74 "Draft a brief refusal and offer a safer alternative."75 ),76 }77 ],78 moderation={"model": "omni-moderation-latest"},79)8081input_moderation = response.moderation.input82output_moderation = response.moderation.output83if input_moderation.type == "error":84 raise RuntimeError(input_moderation.message)85if output_moderation.type == "error":86 raise RuntimeError(output_moderation.message)8788print(input_moderation.flagged)89print(output_moderation.flagged)90```9192```go93package main9495import (96 "context"97 "errors"98 "fmt"99100 "github.com/openai/openai-go/v3"101 "github.com/openai/openai-go/v3/responses"102)103104func main() {105 client := openai.NewClient()106107 response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{108 Model: "gpt-6-astra",109 Input: responses.ResponseNewParamsInputUnion{110 OfString: openai.String("A user asks for instructions to make a harmful weapon. Draft a brief refusal and offer a safer alternative."),111 },112 Moderation: responses.ResponseNewParamsModeration{113 Model: "omni-moderation-latest",114 },115 })116 if err != nil {117 panic(err)118 }119120 switch inputModeration := response.Moderation.Input.AsAny().(type) {121 case responses.ResponseModerationInputModerationResult:122 fmt.Println(inputModeration.Flagged)123 case responses.ResponseModerationInputError:124 panic(errors.New(inputModeration.Message))125 default:126 panic("unexpected input moderation result")127 }128 switch outputModeration := response.Moderation.Output.AsAny().(type) {129 case responses.ResponseModerationOutputModerationResult:130 fmt.Println(outputModeration.Flagged)131 case responses.ResponseModerationOutputError:132 panic(errors.New(outputModeration.Message))133 default:134 panic("unexpected output moderation result")135 }136}137```138139```java140import com.openai.client.OpenAIClient;141import com.openai.client.okhttp.OpenAIOkHttpClient;142import com.openai.core.JsonValue;143import com.openai.models.responses.ResponseCreateParams;144import java.util.ArrayList;145import java.util.List;146import java.util.Map;147148ResponseCreateParams params =149 ResponseCreateParams.builder()150 .model("gpt-6-astra")151 .input(152 "A user asks for instructions to make a harmful weapon. Draft a brief refusal and offer a safer alternative.")153 .putAdditionalBodyProperty(154 "moderation", JsonValue.from(Map.of("model", "omni-moderation-latest")))155 .build();156157var response = client.responses().create(params);158var moderation =159 response160 .moderation()161 .orElseThrow(162 () -> new IllegalStateException("The response did not include moderation results"));163List<Boolean> flags = new ArrayList<>();164165var input = moderation.input();166if (input.isError()) {167 throw new IllegalStateException(input.asError().message());168}169if (!input.isModerationResult()) {170 throw new IllegalStateException("Missing input moderation flag");171}172flags.add(input.asModerationResult().flagged());173174var output = moderation.output();175if (output.isError()) {176 throw new IllegalStateException(output.asError().message());177}178if (!output.isModerationResult()) {179 throw new IllegalStateException("Missing output moderation flag");180}181flags.add(output.asModerationResult().flagged());182183flags.forEach(System.out::println);184```185186```ruby187require "openai"188189client = OpenAI::Client.new190191response = client.responses.create(192 model: "gpt-6-astra",193 input: "A user asks for instructions to make a harmful weapon. Draft a brief refusal and offer a safer alternative.",194 moderation: { model: "omni-moderation-latest" }195)196197puts(response.moderation)198```199200201The Responses API returns an input `moderation_result` object at `response.moderation.input` and an output `moderation_result` object at `response.moderation.output`.202203204205206207Inline moderation results use the same category fields as a standalone moderation result. Start with `flagged` for a first-pass decision, then inspect `categories` and `category_scores` for logging, routing, audit trails, or human-review queues. A refusal or other safety-aware response can still trigger a flag if it discusses harmful content. Treat moderation scores as signals for your application's policy, not as an automatic blocking decision.208209Check the moderation result type before you read scores if your application needs to handle moderation failures. If a moderation step can't complete, the corresponding input or output moderation field can contain an error instead of moderation scores.210211For tool-calling requests, moderation covers tool-call arguments and tool outputs when they appear in conversation content. It doesn't cover tool names, tool descriptions, tool schemas, or response-format schemas.212213If you stream a generated response, moderation scores arrive after the full generated output is available. They aren't included with partial output deltas.214215## Classify standalone inputs216217Use the [moderation endpoint](https://developers.openai.com/api/reference/resources/moderations) to classify text or image inputs without generating a model response. The tabs below show how to use the [OpenAI libraries](https://developers.openai.com/api/docs/libraries) and the [`omni-moderation-latest` model](https://developers.openai.com/api/docs/models#moderation):218219220221Moderate text inputs222223 224225Get classification information for a text input226227```javascript228import OpenAI from "openai";229const openai = new OpenAI();230231const moderation = await openai.moderations.create({232 model: "omni-moderation-latest",233 input: "...text to classify goes here...",234});235236console.log(moderation);237```238239```python240from openai import OpenAI241242client = OpenAI()243244response = client.moderations.create(245 model="omni-moderation-latest",246 input="...text to classify goes here...",247)248249print(response)250```251252```go253package main254255import (256 "context"257 "fmt"258259 "github.com/openai/openai-go/v3"260)261262func main() {263 client := openai.NewClient()264265 moderation, err := client.Moderations.New(context.Background(), openai.ModerationNewParams{266 Model: openai.ModerationModelOmniModerationLatest,267 Input: openai.ModerationNewParamsInputUnion{268 OfString: openai.String("Text to classify goes here."),269 },270 })271 if err != nil {272 panic(err)273 }274275 fmt.Println(moderation.Results[0].Flagged)276}277```278279```java280import com.openai.client.OpenAIClient;281import com.openai.client.okhttp.OpenAIOkHttpClient;282import com.openai.models.moderations.ModerationCreateParams;283284var moderation =285 client286 .moderations()287 .create(288 ModerationCreateParams.builder()289 .model("omni-moderation-latest")290 .input("Text to classify goes here.")291 .build());292293System.out.println(moderation.results().get(0).flagged());294```295296```csharp297using OpenAI.Moderations;298299string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;300string model = "omni-moderation-latest";301ModerationClient client = new(model, key);302303ModerationResult result = await client.ClassifyTextAsync(304 "Text to classify goes here."305);306307Console.WriteLine($"Flagged: {result.Flagged}");308Console.WriteLine(309 $"Violence: {result.Violence.Flagged}; score: {result.Violence.Score:F3}"310);311```312313```ruby314require "openai"315316client = OpenAI::Client.new317318moderation = client.moderations.create(319 model: OpenAI::Models::ModerationModel::OMNI_MODERATION_LATEST,320 input: "Text to classify goes here."321)322323puts(moderation.results.fetch(0).flagged)324```325326```bash327curl https://api.openai.com/v1/moderations \328 -X POST \329 -H "Content-Type: application/json" \330 -H "Authorization: Bearer $OPENAI_API_KEY" \331 -d '{332 "model": "omni-moderation-latest",333 "input": "...text to classify goes here..."334 }'335```336337338 339340 341342 343Moderate images and text344345 346347Get classification information for image and text input348349```javascript350import OpenAI from "openai";351const openai = new OpenAI();352353const moderation = await openai.moderations.create({354 model: "omni-moderation-latest",355 input: [356 { type: "text", text: "...text to classify goes here..." },357 {358 type: "image_url",359 image_url: {360 url: "https://example.com/image.png",361 // You can also use a Base64 encoded image URL.362 // url: "data:image/jpeg;base64,abcdefg...",363 },364 },365 ],366});367368console.log(moderation);369```370371```python372from openai import OpenAI373374client = OpenAI()375376response = client.moderations.create(377 model="omni-moderation-latest",378 input=[379 {"type": "text", "text": "...text to classify goes here..."},380 {381 "type": "image_url",382 "image_url": {383 "url": "https://example.com/image.png",384 # You can also use a Base64 encoded image URL.385 # "url": "data:image/jpeg;base64,abcdefg..."386 },387 },388 ],389)390391print(response)392```393394```go395package main396397import (398 "context"399 "fmt"400401 "github.com/openai/openai-go/v3"402)403404func main() {405 client := openai.NewClient()406407 moderation, err := client.Moderations.New(context.Background(), openai.ModerationNewParams{408 Model: openai.ModerationModelOmniModerationLatest,409 Input: openai.ModerationNewParamsInputUnion{410 OfModerationMultiModalArray: []openai.ModerationMultiModalInputUnionParam{411 openai.ModerationMultiModalInputParamOfText("Text to classify goes here."),412 openai.ModerationMultiModalInputParamOfImageURL(openai.ModerationImageURLInputImageURLParam{413 URL: "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg",414 }),415 },416 },417 })418 if err != nil {419 panic(err)420 }421422 fmt.Println(moderation.Results[0].Flagged)423}424```425426```java427import com.openai.client.OpenAIClient;428import com.openai.client.okhttp.OpenAIOkHttpClient;429import com.openai.models.moderations.ModerationCreateParams;430import com.openai.models.moderations.ModerationImageUrlInput;431import com.openai.models.moderations.ModerationMultiModalInput;432import com.openai.models.moderations.ModerationTextInput;433import java.util.List;434435var moderation =436 client437 .moderations()438 .create(439 ModerationCreateParams.builder()440 .model("omni-moderation-latest")441 .inputOfModerationMultiModalArray(442 List.of(443 ModerationMultiModalInput.ofText(444 ModerationTextInput.builder()445 .text("Text to classify goes here.")446 .build()),447 ModerationMultiModalInput.ofImageUrl(448 ModerationImageUrlInput.builder()449 .imageUrl(450 ModerationImageUrlInput.ImageUrl.builder()451 .url(452 "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg")453 .build())454 .build())))455 .build());456457System.out.println(moderation.results().get(0).flagged());458```459460```csharp461using OpenAI.Moderations;462#pragma warning disable OPENAI001463464string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;465string model = "omni-moderation-latest";466ModerationClient client = new(model, key);467468ModerationResult result = await client.ClassifyInputsAsync(469 [470 ModerationInputPart.CreateTextPart("Text to classify goes here."),471 ModerationInputPart.CreateImagePart(472 new Uri(473 "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg"474 )475 ),476 ]477);478479Console.WriteLine($"Flagged: {result.Flagged}");480Console.WriteLine(481 $"Violence: {result.Violence.Flagged}; score: {result.Violence.Score:F3}; inputs: {result.Violence.ApplicableInputKinds}"482);483```484485```ruby486require "openai"487488client = OpenAI::Client.new489490moderation = client.moderations.create(491 model: OpenAI::Models::ModerationModel::OMNI_MODERATION_LATEST,492 input: [493 {494 type: :text,495 text: "Text to classify goes here."496 },497 {498 type: :image_url,499 image_url: {500 url: "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg"501 }502 }503 ]504)505506puts(moderation.results.fetch(0).flagged)507```508509```bash510curl https://api.openai.com/v1/moderations \511 -X POST \512 -H "Content-Type: application/json" \513 -H "Authorization: Bearer $OPENAI_API_KEY" \514 -d '{515 "model": "omni-moderation-latest",516 "input": [517 { "type": "text", "text": "...text to classify goes here..." },518 {519 "type": "image_url",520 "image_url": {521 "url": "https://example.com/image.png"522 }523 }524 ]525 }'526```527528529530## Understand moderation results531532Here's a full example output for an image from a single frame of a war movie. The model identifies indicators of violence in the image, with a `violence` category score greater than 0.8.533534```json535{536 "id": "modr-970d409ef3bef3b70c73d8232df86e7d",537 "model": "omni-moderation-latest",538 "results": [539 {540 "flagged": true,541 "categories": {542 "sexual": false,543 "sexual/minors": false,544 "harassment": false,545 "harassment/threatening": false,546 "hate": false,547 "hate/threatening": false,548 "illicit": false,549 "illicit/violent": false,550 "self-harm": false,551 "self-harm/intent": false,552 "self-harm/instructions": false,553 "violence": true,554 "violence/graphic": false555 },556 "category_scores": {557 "sexual": 2.34135824776394e-7,558 "sexual/minors": 1.6346470245419304e-7,559 "harassment": 0.0011643905680426018,560 "harassment/threatening": 0.0022121340080906377,561 "hate": 3.1999824407395835e-7,562 "hate/threatening": 2.4923252458203563e-7,563 "illicit": 0.0005227032493135171,564 "illicit/violent": 3.682979260160596e-7,565 "self-harm": 0.0011175734280627694,566 "self-harm/intent": 0.0006264858507989037,567 "self-harm/instructions": 7.368592981140821e-8,568 "violence": 0.8599265510337075,569 "violence/graphic": 0.37701736389561064570 },571 "category_applied_input_types": {572 "sexual": ["image"],573 "sexual/minors": [],574 "harassment": [],575 "harassment/threatening": [],576 "hate": [],577 "hate/threatening": [],578 "illicit": [],579 "illicit/violent": [],580 "self-harm": ["image"],581 "self-harm/intent": ["image"],582 "self-harm/instructions": ["image"],583 "violence": ["image"],584 "violence/graphic": ["image"]585 }586 }587 ]588}589```590591The JSON response includes fields that describe which categories are present in the input and the model's confidence in each category.592593<table>594 <tr>595 <th>Output category</th>596 <th>Description</th>597 </tr>598 <tr>599 <td>`flagged`</td>600 <td>601 Set to `true` if the model classifies the content as potentially harmful,602 `false` otherwise.603 </td>604 </tr>605 <tr>606 <td>`categories`</td>607 <td>608 Contains a dictionary of per-category violation flags. For each category,609 the value is `true` if the model flags the corresponding category as610 violated, `false` otherwise.611 </td>612 </tr>613 <tr>614 <td>`category_scores`</td>615 <td>616 Contains a dictionary of per-category scores. Each score represents the617 model's confidence that the input contains content in the category. The618 value is between 0 and 1, where higher values denote higher confidence.619 </td>620 </tr>621 <tr>622 <td>`category_applied_input_types`</td>623 <td>624 Contains the input types that the category score applies to. For example,625 if the `violence/graphic` category applies to both image and text inputs,626 the `violence/graphic` property is set to `["image", "text"]`.627 </td>628 </tr>629</table>630631We plan to continuously upgrade the moderation endpoint's underlying model.632 Therefore, custom policies that rely on `category_scores` may need633 recalibration over time.634635## Review supported categories636637The table below describes the content categories that the moderation endpoint can detect and the input types that each category supports.638639Categories marked as "Text only" do not support image inputs. If you send only640 images (without accompanying text) to the `omni-moderation-latest` model, it641 will return a score of 0 for these unsupported categories. Image files are642 limited to 20 MB.643644<table>645 <tr>646 <th>647 **Category**648 </th>649 <th>650 **Description**651 </th>652 <th>653 **Inputs**654 </th>655 </tr>656 <tr>657 <td>`harassment`</td>658 <td>659 Content that expresses, incites, or promotes harassing language towards660 any target.661 </td>662 <td>Text only</td>663 </tr>664 <tr>665 <td>`harassment/threatening`</td>666 <td>667 Harassment content that also includes violence or serious harm towards any668 target.669 </td>670 <td>Text only</td>671 </tr>672 <tr>673 <td>`hate`</td>674 <td>675 Content that expresses, incites, or promotes hate based on race, gender,676 ethnicity, religion, nationality, sexual orientation, disability status,677 or caste. Hateful content aimed at non-protected groups (e.g., chess678 players) is harassment.679 </td>680 <td>Text only</td>681 </tr>682 <tr>683 <td>`hate/threatening`</td>684 <td>685 Hateful content that also includes violence or serious harm towards the686 targeted group based on race, gender, ethnicity, religion, nationality,687 sexual orientation, disability status, or caste.688 </td>689 <td>Text only</td>690 </tr>691 <tr>692 <td>`illicit`</td>693 <td>694 Content that gives advice or instruction on how to commit illicit acts. A695 phrase like "how to shoplift" would fit this category.696 </td>697 <td>Text only</td>698 </tr>699 <tr>700 <td>`illicit/violent`</td>701 <td>702 The same types of content flagged by the `illicit` category, but also703 includes references to violence or procuring a weapon.704 </td>705 <td>Text only</td>706 </tr>707 <tr>708 <td>`self-harm`</td>709 <td>710 Content that promotes, encourages, or depicts acts of self-harm, such as711 suicide, cutting, and eating disorders.712 </td>713 <td>Text and images</td>714 </tr>715 <tr>716 <td>`self-harm/intent`</td>717 <td>718 Content where the speaker expresses that they are engaging or intend to719 engage in acts of self-harm, such as suicide, cutting, and eating720 disorders.721 </td>722 <td>Text and images</td>723 </tr>724 <tr>725 <td>`self-harm/instructions`</td>726 <td>727 Content that encourages performing acts of self-harm, such as suicide,728 cutting, and eating disorders, or that gives instructions or advice on how729 to commit such acts.730 </td>731 <td>Text and images</td>732 </tr>733 <tr>734 <td>`sexual`</td>735 <td>736 Content meant to arouse sexual excitement, such as the description of737 sexual activity, or that promotes sexual services (excluding sex education738 and wellness).739 </td>740 <td>Text and images</td>741 </tr>742 <tr>743 <td>`sexual/minors`</td>744 <td>745 Sexual content that includes an individual who is under 18 years old.746 </td>747 <td>Text only</td>748 </tr>749 <tr>750 <td>`violence`</td>751 <td>Content that depicts death, violence, or physical injury.</td>752 <td>Text and images</td>753 </tr>754 <tr>755 <td>`violence/graphic`</td>756 <td>757 Content that depicts death, violence, or physical injury in graphic758 detail.759 </td>760 <td>Text and images</td>761 </tr>762</table>