Repository Public
1---2title: Content moderation3url: https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation4description: Content moderation is a critical aspect of maintaining a safe, respectful, and productive environment in digital applications. This guide discusses how Claude can be used to moderate content within your digital application.5---67> Visit the [content moderation cookbook](https://platform.claude.com/cookbook/misc-building-moderation-filter) to see an example content moderation implementation using Claude.89<Tip>10 This guide is focused on moderating user-generated content within your application. If you're looking for guidance on moderating interactions with Claude, refer to 1112 [Mitigate jailbreaks and prompt injections](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks)1314 .15</Tip>1617## Before building with Claude1819### Decide whether to use Claude for content moderation2021Here are some key indicators that you should use an LLM like Claude instead of a traditional ML or rules-based approach for content moderation:2223<AccordionGroup>24 <Accordion title="You want a cost-effective and rapid implementation">25 Traditional ML methods require significant engineering resources, ML expertise, and infrastructure costs. Human moderation systems incur even higher costs. With Claude, you can have a sophisticated moderation system operational in significantly less time and at a much lower cost.26 </Accordion>2728 <Accordion title="You want both semantic understanding and quick decisions">29 Traditional ML approaches, such as bag-of-words models or simple pattern matching, often struggle to understand the tone, intent, and context of the content. While human moderation systems excel at understanding semantic meaning, they require time for content to be reviewed. Claude addresses both needs by combining semantic understanding with the ability to deliver moderation decisions quickly.30 </Accordion>3132 <Accordion title="You need consistent policy decisions">33 By leveraging its advanced reasoning capabilities, Claude can interpret and apply complex moderation guidelines uniformly. This consistency helps ensure fair treatment of all content, reducing the risk of inconsistent or biased moderation decisions that can undermine user trust.34 </Accordion>3536 <Accordion title="Your moderation policies are likely to change or evolve over time">37 Once a traditional ML approach has been established, changing it is a laborious and data-intensive undertaking. On the other hand, as your product or customer needs evolve, Claude can easily adapt to changes or additions to moderation policies without extensive relabeling of training data.38 </Accordion>3940 <Accordion title="You require interpretable reasoning for your moderation decisions">41 If you want to provide users or regulators with clear explanations behind moderation decisions, Claude can generate detailed and coherent justifications. This transparency is important for building trust and ensuring accountability in content moderation practices.42 </Accordion>4344 <Accordion title="You need multilingual support without maintaining separate models">45 Traditional ML approaches typically require separate models or extensive translation processes for each supported language. Human moderation requires hiring a workforce fluent in each supported language. Claude’s multilingual capabilities allow it to classify tickets in various languages without the need for separate models or extensive translation processes, streamlining moderation for global customer bases.46 </Accordion>4748 <Accordion title="You require multimodal support">49 Claude's multimodal capabilities allow it to analyze and interpret content across both text and images. This makes it a versatile tool for comprehensive content moderation in environments where different media types need to be evaluated together.50 </Accordion>51</AccordionGroup>5253<Note>54 All Claude models are trained with built-in safety behaviors. This may result in Claude moderating content deemed particularly dangerous (in line with the 5556 [Acceptable Use Policy](https://www.anthropic.com/legal/aup)5758 ), regardless of the prompt used. For example, an adult website that wants to allow users to post explicit sexual content may find that Claude still flags explicit content as requiring moderation, even if they specify in their prompt not to moderate explicit sexual content. Consider reviewing the AUP in advance of building a moderation solution.59</Note>6061### Generate examples of content to moderate6263Before developing a content moderation solution, first create examples of content that should be flagged and content that should not be flagged. Ensure that you include edge cases and challenging scenarios that may be difficult for a content moderation system to handle effectively. Afterward, review your examples to create a well-defined list of moderation categories. For instance, the examples generated by a social media platform might include the following:6465<CodeGroup exclude="shell">66 ```python Python67 client = anthropic.Anthropic()6869 allowed_user_comments = [70 "This movie was great, I really enjoyed it. The main actor really killed it!",71 "I hate Mondays.",72 "It is a great time to invest in gold!",73 ]7475 disallowed_user_comments = [76 "Delete this post now or you better hide. I am coming after you and your family.",77 "Stay away from the 5G cellphones!! They are using 5G to control you.",78 "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",79 ]8081 # Sample user comments to test the content moderation82 user_comments = allowed_user_comments + disallowed_user_comments8384 # Categories considered unsafe for content moderation85 unsafe_categories = [86 "Child Exploitation",87 "Conspiracy Theories",88 "Hate",89 "Indiscriminate Weapons",90 "Intellectual Property",91 "Non-Violent Crimes",92 "Privacy",93 "Self-Harm",94 "Sex Crimes",95 "Sexual Content",96 "Specialized Advice",97 "Violent Crimes",98 ]99 ```100101 ```typescript TypeScript102 const client = new Anthropic();103104 const allowedUserComments = [105 "This movie was great, I really enjoyed it. The main actor really killed it!",106 "I hate Mondays.",107 "It is a great time to invest in gold!"108 ];109110 const disallowedUserComments = [111 "Delete this post now or you better hide. I am coming after you and your family.",112 "Stay away from the 5G cellphones!! They are using 5G to control you.",113 "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"114 ];115116 // Sample user comments to test the content moderation117 const userComments = [...allowedUserComments, ...disallowedUserComments];118119 // Categories considered unsafe for content moderation120 const unsafeCategories = [121 "Child Exploitation",122 "Conspiracy Theories",123 "Hate",124 "Indiscriminate Weapons",125 "Intellectual Property",126 "Non-Violent Crimes",127 "Privacy",128 "Self-Harm",129 "Sex Crimes",130 "Sexual Content",131 "Specialized Advice",132 "Violent Crimes"133 ];134 ```135136 ```csharp C#137 var client = new AnthropicClient();138139 string[] allowedUserComments =140 [141 "This movie was great, I really enjoyed it. The main actor really killed it!",142 "I hate Mondays.",143 "It is a great time to invest in gold!",144 ];145146 string[] disallowedUserComments =147 [148 "Delete this post now or you better hide. I am coming after you and your family.",149 "Stay away from the 5G cellphones!! They are using 5G to control you.",150 "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",151 ];152153 // Sample user comments to test the content moderation154 string[] userComments = [.. allowedUserComments, .. disallowedUserComments];155156 // Categories considered unsafe for content moderation157 string[] unsafeCategories =158 [159 "Child Exploitation",160 "Conspiracy Theories",161 "Hate",162 "Indiscriminate Weapons",163 "Intellectual Property",164 "Non-Violent Crimes",165 "Privacy",166 "Self-Harm",167 "Sex Crimes",168 "Sexual Content",169 "Specialized Advice",170 "Violent Crimes",171 ];172 ```173174 ```go Go175 var client = anthropic.NewClient()176177 var allowedUserComments = []string{178 "This movie was great, I really enjoyed it. The main actor really killed it!",179 "I hate Mondays.",180 "It is a great time to invest in gold!",181 }182183 var disallowedUserComments = []string{184 "Delete this post now or you better hide. I am coming after you and your family.",185 "Stay away from the 5G cellphones!! They are using 5G to control you.",186 "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",187 }188189 // Sample user comments to test the content moderation190 var userComments = slices.Concat(allowedUserComments, disallowedUserComments)191192 // Categories considered unsafe for content moderation193 var unsafeCategories = []string{194 "Child Exploitation",195 "Conspiracy Theories",196 "Hate",197 "Indiscriminate Weapons",198 "Intellectual Property",199 "Non-Violent Crimes",200 "Privacy",201 "Self-Harm",202 "Sex Crimes",203 "Sexual Content",204 "Specialized Advice",205 "Violent Crimes",206 }207208 ```209210 ```java Java211 final AnthropicClient client = AnthropicOkHttpClient.fromEnv();212213 final List<String> allowedUserComments = List.of(214 "This movie was great, I really enjoyed it. The main actor really killed it!",215 "I hate Mondays.",216 "It is a great time to invest in gold!");217218 final List<String> disallowedUserComments = List.of(219 "Delete this post now or you better hide. I am coming after you and your family.",220 "Stay away from the 5G cellphones!! They are using 5G to control you.",221 "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!");222223 // Sample user comments to test the content moderation224 final List<String> userComments =225 Stream.concat(allowedUserComments.stream(), disallowedUserComments.stream()).toList();226227 // Categories considered unsafe for content moderation228 final List<String> unsafeCategories = List.of(229 "Child Exploitation",230 "Conspiracy Theories",231 "Hate",232 "Indiscriminate Weapons",233 "Intellectual Property",234 "Non-Violent Crimes",235 "Privacy",236 "Self-Harm",237 "Sex Crimes",238 "Sexual Content",239 "Specialized Advice",240 "Violent Crimes");241 ```242243 ```php PHP244 $client = new Client();245246 $allowedUserComments = [247 'This movie was great, I really enjoyed it. The main actor really killed it!',248 'I hate Mondays.',249 'It is a great time to invest in gold!',250 ];251252 $disallowedUserComments = [253 'Delete this post now or you better hide. I am coming after you and your family.',254 'Stay away from the 5G cellphones!! They are using 5G to control you.',255 'Congratulations! You have won a $1,000 gift card. Click here to claim your prize!',256 ];257258 // Sample user comments to test the content moderation259 $userComments = [...$allowedUserComments, ...$disallowedUserComments];260261 // Categories considered unsafe for content moderation262 $unsafeCategories = [263 'Child Exploitation',264 'Conspiracy Theories',265 'Hate',266 'Indiscriminate Weapons',267 'Intellectual Property',268 'Non-Violent Crimes',269 'Privacy',270 'Self-Harm',271 'Sex Crimes',272 'Sexual Content',273 'Specialized Advice',274 'Violent Crimes',275 ];276 ```277278 ```ruby Ruby279 CLIENT = Anthropic::Client.new280281 ALLOWED_USER_COMMENTS = [282 "This movie was great, I really enjoyed it. The main actor really killed it!",283 "I hate Mondays.",284 "It is a great time to invest in gold!"285 ]286287 DISALLOWED_USER_COMMENTS = [288 "Delete this post now or you better hide. I am coming after you and your family.",289 "Stay away from the 5G cellphones!! They are using 5G to control you.",290 "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"291 ]292293 # Sample user comments to test the content moderation294 USER_COMMENTS = ALLOWED_USER_COMMENTS + DISALLOWED_USER_COMMENTS295296 # Categories considered unsafe for content moderation297 UNSAFE_CATEGORIES = [298 "Child Exploitation",299 "Conspiracy Theories",300 "Hate",301 "Indiscriminate Weapons",302 "Intellectual Property",303 "Non-Violent Crimes",304 "Privacy",305 "Self-Harm",306 "Sex Crimes",307 "Sexual Content",308 "Specialized Advice",309 "Violent Crimes"310 ]311 ```312</CodeGroup>313314Effectively moderating these examples requires a nuanced understanding of language. In the comment, `This movie was great, I really enjoyed it. The main actor really killed it!`, the content moderation system needs to recognize that "killed it" is a metaphor, not an indication of actual violence. Conversely, despite the lack of explicit mentions of violence, the comment `Delete this post now or you better hide. I am coming after you and your family.` should be flagged by the content moderation system.315316The unsafe categories can be customized to fit your specific needs. For example, if you want to prevent minors from creating content on your website, you could add "Underage Posting" to the categories.317318***319320## How to moderate content using Claude321322### Select the right Claude model323324When selecting a model, it’s important to consider the size of your data. If costs are a concern, a smaller model such as Claude Haiku 4.5 is an excellent choice because of its cost-effectiveness. The following is an estimate of the cost to moderate text for a social media platform that receives one billion posts per month:325326* **Content size**327328 * Posts per month: 1B329 * Characters per post: 100330 * Total characters: 100B331332* **Estimated tokens**333334 * Input tokens: 28.6B (assuming 1 token per 3.5 characters)335 * Percentage of messages flagged: 3%336 * Output tokens per flagged message: 50337 * Total output tokens: 1.5B338339* **Claude Haiku 4.5 estimated cost**340341 * Input token cost: 28,600 MTok \* $1.00/MTok = $28,600 USD342 * Output token cost: 1,500 MTok \* $5.00/MTok = $7,500 USD343 * Monthly cost: $28,600 + $7,500 = $36,100 USD344345* **Claude Opus 5 estimated cost**346347 * Input token cost: 28,600 MTok \* $5.00/MTok = $143,000 USD348 * Output token cost: 1,500 MTok \* $25.00/MTok = $37,500 USD349 * Monthly cost: $143,000 + $37,500 = $180,500 USD350351* **Claude Opus 4.8 estimated cost**352353 * Input token cost: 28,600 MTok \* $5.00/MTok = $143,000 USD354 * Output token cost: 1,500 MTok \* $25.00/MTok = $37,500 USD355 * Monthly cost: $143,000 + $37,500 = $180,500 USD356357<Tip>358 Actual costs may differ from these estimates. These estimates are based on the prompt highlighted in the section on 359360 [batch processing](https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation#consider-batch-processing)361362 . Output tokens can be reduced even further by removing the 363364 `explanation`365366 field from the response.367</Tip>368369### Build a strong prompt370371To use Claude for content moderation, Claude must understand the moderation requirements of your application. Start by writing a prompt that allows you to define your moderation needs:372373<CodeGroup exclude="shell">374 ```python Python375 def moderate_message(message, unsafe_categories):376 # Convert the unsafe categories into a string, with each category on a new line377 unsafe_category_str = "\n".join(unsafe_categories)378379 # Construct the prompt for Claude, including the message and unsafe categories380 assessment_prompt = f"""381 Determine whether the following message warrants moderation,382 based on the unsafe categories outlined below.383384 Message:385 <message>{message}</message>386387 Unsafe Categories:388 <categories>389 {unsafe_category_str}390 </categories>391392 Respond with ONLY a JSON object, using the format below:393 {{394 "violation": <Boolean field denoting whether the message should be moderated>,395 "categories": [Comma-separated list of violated categories],396 "explanation": [Optional. Only include if there is a violation.]397 }}398 Do not include markdown formatting or code fences in your response."""399400 # Send the request to Claude for content moderation401 response = client.messages.create(402 model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs403 max_tokens=200,404 messages=[{"role": "user", "content": assessment_prompt}],405 )406407 # Parse the JSON response from Claude408 text_block = next(block for block in response.content if block.type == "text")409 assessment = json.loads(text_block.text)410411 # Extract the violation status from the assessment412 contains_violation = assessment["violation"]413414 # If there's a violation, get the categories and explanation; otherwise, use empty defaults415 violated_categories = assessment.get("categories", []) if contains_violation else []416 explanation = assessment.get("explanation") if contains_violation else None417418 return contains_violation, violated_categories, explanation419420421 # Process each comment and print the results422 for comment in user_comments:423 print(f"\nComment: {comment}")424 violation, violated_categories, explanation = moderate_message(425 comment, unsafe_categories426 )427428 if violation:429 print(f"Violated Categories: {', '.join(violated_categories)}")430 print(f"Explanation: {explanation}")431 else:432 print("No issues detected.")433 ```434435 ```typescript TypeScript436 // Shape of the JSON assessment Claude returns437 interface ModerationAssessment {438 violation: boolean;439 categories?: string[];440 explanation?: string;441 }442443 async function moderateMessage(444 message: string,445 unsafeCategories: string[]446 ): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {447 // Convert the unsafe categories into a string, with each category on a new line448 const unsafeCategoryStr = unsafeCategories.join("\n");449450 // Construct the prompt for Claude, including the message and unsafe categories451 const assessmentPrompt = `452 Determine whether the following message warrants moderation,453 based on the unsafe categories outlined below.454455 Message:456 <message>${message}</message>457458 Unsafe Categories:459 <categories>460 ${unsafeCategoryStr}461 </categories>462463 Respond with ONLY a JSON object, using the format below:464 {465 "violation": <Boolean field denoting whether the message should be moderated>,466 "categories": [Comma-separated list of violated categories],467 "explanation": [Optional. Only include if there is a violation.]468 }469 Do not include markdown formatting or code fences in your response.`;470471 // Send the request to Claude for content moderation472 const response = await client.messages.create({473 model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs474 max_tokens: 200,475 messages: [{ role: "user", content: assessmentPrompt }]476 });477478 // Parse the JSON response from Claude479 const textBlock = response.content.find((block) => block.type === "text");480 if (!textBlock) {481 throw new Error("Expected a text block in the response");482 }483 const assessment: ModerationAssessment = JSON.parse(textBlock.text);484485 // Extract the violation status from the assessment486 const containsViolation = assessment.violation;487488 // If there's a violation, get the categories and explanation; otherwise, use empty defaults489 const violatedCategories = containsViolation ? assessment.categories ?? [] : [];490 const explanation = containsViolation ? assessment.explanation : undefined;491492 return { violation: containsViolation, violatedCategories, explanation };493 }494495 // Process each comment and print the results496 for (const comment of userComments) {497 console.log(`\nComment: ${comment}`);498 const { violation, violatedCategories, explanation } = await moderateMessage(499 comment,500 unsafeCategories501 );502503 if (violation) {504 console.log(`Violated Categories: ${violatedCategories.join(", ")}`);505 console.log(`Explanation: ${explanation}`);506 } else {507 console.log("No issues detected.");508 }509 }510 ```511512 ```csharp C#513 async Task<(bool ContainsViolation, List<string> ViolatedCategories, string? Explanation)> ModerateMessage(514 string message,515 IReadOnlyList<string> categories516 )517 {518 // Convert the unsafe categories into a string, with each category on a new line519 var unsafeCategoryText = string.Join("\n", categories);520521 // Construct the prompt for Claude, including the message and unsafe categories522 var assessmentPrompt = $$"""523524 Determine whether the following message warrants moderation,525 based on the unsafe categories outlined below.526527 Message:528 <message>{{message}}</message>529530 Unsafe Categories:531 <categories>532 {{unsafeCategoryText}}533 </categories>534535 Respond with ONLY a JSON object, using the format below:536 {537 "violation": <Boolean field denoting whether the message should be moderated>,538 "categories": [Comma-separated list of violated categories],539 "explanation": [Optional. Only include if there is a violation.]540 }541 Do not include markdown formatting or code fences in your response.542 """;543544 // Send the request to Claude for content moderation545 var response = await client.Messages.Create(546 new()547 {548 Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs549 MaxTokens = 200,550 Messages = [new() { Role = Role.User, Content = assessmentPrompt }],551 }552 );553554 // Narrow the first content block to a text block, then parse Claude's JSON response555 if (!response.Content[0].TryPickText(out var textBlock))556 {557 throw new InvalidOperationException("Expected a text response from Claude.");558 }559 var assessment = JsonNode.Parse(textBlock.Text)!;560561 // Extract the violation status from the assessment562 var containsViolation = assessment["violation"]!.GetValue<bool>();563564 // If there's a violation, get the categories and explanation; otherwise, use empty defaults565 List<string> violatedCategories = containsViolation566 ? assessment["categories"]?.AsArray().Select(category => category!.GetValue<string>()).ToList() ?? []567 : [];568 var explanation = containsViolation ? assessment["explanation"]?.GetValue<string>() : null;569570 return (containsViolation, violatedCategories, explanation);571 }572573 // Process each comment and print the results574 foreach (var comment in userComments)575 {576 Console.WriteLine($"\nComment: {comment}");577 var (violation, violatedCategories, explanation) = await ModerateMessage(comment, unsafeCategories);578579 if (violation)580 {581 Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");582 Console.WriteLine($"Explanation: {explanation}");583 }584 else585 {586 Console.WriteLine("No issues detected.");587 }588 }589 ```590591 ```go Go592 func moderateMessage(message string, unsafeCategories []string) (bool, []string, string) {593 // Convert the unsafe categories into a string, with each category on a new line594 unsafeCategoryStr := strings.Join(unsafeCategories, "\n")595596 // Construct the prompt for Claude, including the message and unsafe categories597 assessmentPrompt := fmt.Sprintf(`598 Determine whether the following message warrants moderation,599 based on the unsafe categories outlined below.600601 Message:602 <message>%s</message>603604 Unsafe Categories:605 <categories>606 %s607 </categories>608609 Respond with ONLY a JSON object, using the format below:610 {611 "violation": <Boolean field denoting whether the message should be moderated>,612 "categories": [Comma-separated list of violated categories],613 "explanation": [Optional. Only include if there is a violation.]614 }615 Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)616617 // Send the request to Claude for content moderation618 response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{619 Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs620 MaxTokens: 200,621 Messages: []anthropic.MessageParam{622 anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),623 },624 })625 if err != nil {626 log.Fatal(err)627 }628629 // Narrow the first content block to a text block before reading its text630 textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)631 if !ok {632 log.Fatalf("expected a text block, got %q", response.Content[0].Type)633 }634635 // Parse the JSON response from Claude636 var assessment struct {637 Violation bool `json:"violation"`638 Categories []string `json:"categories"`639 Explanation string `json:"explanation"`640 }641 if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {642 log.Fatal(err)643 }644645 // If there's a violation, return the categories and explanation; otherwise, use empty defaults646 if !assessment.Violation {647 return false, nil, ""648 }649 return true, assessment.Categories, assessment.Explanation650 }651652 // moderateAllComments processes each comment and prints the results.653 func moderateAllComments() {654 for _, comment := range userComments {655 fmt.Printf("\nComment: %s\n", comment)656 violation, violatedCategories, explanation := moderateMessage(comment, unsafeCategories)657658 if violation {659 fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))660 fmt.Printf("Explanation: %s\n", explanation)661 } else {662 fmt.Println("No issues detected.")663 }664 }665 }666667 ```668669 ```java Java670 record ModerationResult(boolean violation, List<String> violatedCategories, String explanation) {}671672 ModerationResult moderateMessage(String message, List<String> unsafeCategories)673 throws JsonProcessingException {674 // Convert the unsafe categories into a string, with each category on a new line675 String unsafeCategoryStr = String.join("\n", unsafeCategories);676677 // Construct the prompt for Claude, including the message and unsafe categories678 String assessmentPrompt = """679680 Determine whether the following message warrants moderation,681 based on the unsafe categories outlined below.682683 Message:684 <message>%s</message>685686 Unsafe Categories:687 <categories>688 %s689 </categories>690691 Respond with ONLY a JSON object, using the format below:692 {693 "violation": <Boolean field denoting whether the message should be moderated>,694 "categories": [Comma-separated list of violated categories],695 "explanation": [Optional. Only include if there is a violation.]696 }697 Do not include markdown formatting or code fences in your response."""698 .formatted(message, unsafeCategoryStr);699700 // Send the request to Claude for content moderation701 Message response = client.messages().create(MessageCreateParams.builder()702 .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs703 .maxTokens(200)704 .addUserMessage(assessmentPrompt)705 .build());706707 // Parse the JSON response from Claude708 String assessmentJson = response.content().stream()709 .flatMap(contentBlock -> contentBlock.text().stream())710 .findFirst()711 .orElseThrow()712 .text();713 ObjectMapper mapper = new ObjectMapper();714 JsonNode assessment = mapper.readTree(assessmentJson);715716 // Extract the violation status from the assessment717 boolean containsViolation = assessment.required("violation").asBoolean();718719 // If there's a violation, get the categories and explanation; otherwise, use empty defaults720 List<String> violatedCategories = containsViolation && assessment.has("categories")721 ? mapper.convertValue(assessment.get("categories"), new TypeReference<List<String>>() {})722 : List.of();723 String explanation = containsViolation && assessment.hasNonNull("explanation")724 ? assessment.get("explanation").asText()725 : null;726727 return new ModerationResult(containsViolation, violatedCategories, explanation);728 }729730 // Process each comment and print the results731 void printModerationResults() throws JsonProcessingException {732 for (String comment : userComments) {733 IO.println("\nComment: " + comment);734 ModerationResult result = moderateMessage(comment, unsafeCategories);735736 if (result.violation()) {737 IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));738 IO.println("Explanation: " + result.explanation());739 } else {740 IO.println("No issues detected.");741 }742 }743 }744 ```745746 ```php PHP747 $moderateMessage = function (string $message, array $unsafeCategories) use ($client): array {748 // Convert the unsafe categories into a string, with each category on a new line749 $unsafeCategoryStr = implode("\n", $unsafeCategories);750751 // Construct the prompt for Claude, including the message and unsafe categories752 $assessmentPrompt = <<<PROMPT753754 Determine whether the following message warrants moderation,755 based on the unsafe categories outlined below.756757 Message:758 <message>{$message}</message>759760 Unsafe Categories:761 <categories>762 {$unsafeCategoryStr}763 </categories>764765 Respond with ONLY a JSON object, using the format below:766 {767 "violation": <Boolean field denoting whether the message should be moderated>,768 "categories": [Comma-separated list of violated categories],769 "explanation": [Optional. Only include if there is a violation.]770 }771 Do not include markdown formatting or code fences in your response.772 PROMPT;773774 // Send the request to Claude for content moderation775 $response = $client->messages->create(776 model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs777 maxTokens: 200,778 messages: [['role' => 'user', 'content' => $assessmentPrompt]],779 );780781 // Parse the JSON response from Claude. The SDK decodes each content block782 // into its concrete class, so find the TextBlock before reading the text.783 $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)784 ?? throw new RuntimeException('Expected a text block in the response.');785 $assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);786787 // Extract the violation status from the assessment788 $containsViolation = $assessment['violation'];789790 // If there's a violation, get the categories and explanation; otherwise, use empty defaults791 $violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];792 $explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;793794 return [$containsViolation, $violatedCategories, $explanation];795 };796797 // Process each comment and print the results798 foreach ($userComments as $comment) {799 echo "\nComment: {$comment}\n";800 [$violation, $violatedCategories, $explanation] = $moderateMessage($comment, $unsafeCategories);801802 if ($violation) {803 echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";804 echo "Explanation: {$explanation}\n";805 } else {806 echo "No issues detected.\n";807 }808 }809 ```810811 ```ruby Ruby812 def moderate_message(message, unsafe_categories)813 # Convert the unsafe categories into a string, with each category on a new line814 unsafe_category_str = unsafe_categories.join("\n")815816 # Construct the prompt for Claude, including the message and unsafe categories817 assessment_prompt = <<~PROMPT.chomp818819 Determine whether the following message warrants moderation,820 based on the unsafe categories outlined below.821822 Message:823 <message>#{message}</message>824825 Unsafe Categories:826 <categories>827 #{unsafe_category_str}828 </categories>829830 Respond with ONLY a JSON object, using the format below:831 {832 "violation": <Boolean field denoting whether the message should be moderated>,833 "categories": [Comma-separated list of violated categories],834 "explanation": [Optional. Only include if there is a violation.]835 }836 Do not include markdown formatting or code fences in your response.837 PROMPT838839 # Send the request to Claude for content moderation840 response = CLIENT.messages.create(841 model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs842 max_tokens: 200,843 messages: [{role: :user, content: assessment_prompt}]844 )845846 # Parse the JSON response from Claude847 text_block = response.content.find { it.type == :text }848 assessment = JSON.parse(text_block.text)849850 # Extract the violation status from the assessment851 contains_violation = assessment["violation"]852853 # If there's a violation, get the categories and explanation; otherwise, use empty defaults854 violated_categories = contains_violation ? assessment.fetch("categories", []) : []855 explanation = contains_violation ? assessment["explanation"] : nil856857 [contains_violation, violated_categories, explanation]858 end859860861 # Process each comment and print the results862 USER_COMMENTS.each do |comment|863 puts "\nComment: #{comment}"864 violation, violated_categories, explanation = moderate_message(comment, UNSAFE_CATEGORIES)865866 if violation867 puts "Violated Categories: #{violated_categories.join(", ")}"868 puts "Explanation: #{explanation}"869 else870 puts "No issues detected."871 end872 end873 ```874</CodeGroup>875876In this example, the `moderate_message` function contains an assessment prompt that includes the unsafe content categories and the message to evaluate. The prompt asks Claude to assess whether the message should be moderated, based on the unsafe categories you defined earlier.877878The model's assessment is then parsed to determine if there is a violation. If there is a violation, Claude also returns a list of violated categories and an explanation as to why the message is unsafe.879880### Evaluate your prompt881882Content moderation is a classification problem. Thus, you can use the same techniques outlined in the [classification cookbook](https://platform.claude.com/cookbook/capabilities-classification-guide) to determine the accuracy of your content moderation system.883884One additional consideration is that instead of treating content moderation as a binary classification problem, you may instead create multiple categories to represent various risk levels. Creating multiple risk levels allows you to adjust the aggressiveness of your moderation. For example, you might want to automatically block user queries that are deemed high risk, while users with many medium risk queries are flagged for human review.885886<CodeGroup exclude="shell">887 ```python Python888 def assess_risk_level(message, unsafe_categories):889 # Convert the unsafe categories into a string, with each category on a new line890 unsafe_category_str = "\n".join(unsafe_categories)891892 # Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions893 assessment_prompt = f"""894 Assess the risk level of the following message,895 based on the unsafe categories listed below.896897 Message:898 <message>{message}</message>899900 Unsafe Categories:901 <categories>902 {unsafe_category_str}903 </categories>904905 Assign a risk level based on your confidence that the user's message should be moderated906 based on the defined unsafe categories:907908 0 - No risk909 1 - Low risk910 2 - Medium risk911 3 - High risk912913 Respond with ONLY a JSON object, using the format below:914 {{915 "risk_level": <Numerical field denoting the risk level>,916 "categories": [Comma-separated list of violated categories],917 "explanation": <Optional. Only include if risk level is greater than 0>918 }}919 Do not include markdown formatting or code fences in your response."""920921 # Send the request to Claude for risk assessment922 response = client.messages.create(923 model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs924 max_tokens=200,925 messages=[{"role": "user", "content": assessment_prompt}],926 )927928 # Parse the JSON response from Claude929 text_block = next(block for block in response.content if block.type == "text")930 assessment = json.loads(text_block.text)931932 # Extract the risk level, violated categories, and explanation from the assessment933 risk_level = assessment["risk_level"]934 violated_categories = assessment["categories"]935 explanation = assessment.get("explanation")936937 return risk_level, violated_categories, explanation938939940 # Process each comment and print the results941 for comment in user_comments:942 print(f"\nComment: {comment}")943 risk_level, violated_categories, explanation = assess_risk_level(944 comment, unsafe_categories945 )946947 print(f"Risk Level: {risk_level}")948 if violated_categories:949 print(f"Violated Categories: {', '.join(violated_categories)}")950 if explanation:951 print(f"Explanation: {explanation}")952 ```953954 ```typescript TypeScript955 // Shape of the JSON risk assessment Claude returns956 interface RiskAssessment {957 risk_level: number;958 categories: string[];959 explanation?: string;960 }961962 async function assessRiskLevel(963 message: string,964 unsafeCategories: string[]965 ): Promise<{ riskLevel: number; violatedCategories: string[]; explanation?: string }> {966 // Convert the unsafe categories into a string, with each category on a new line967 const unsafeCategoryStr = unsafeCategories.join("\n");968969 // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions970 const assessmentPrompt = `971 Assess the risk level of the following message,972 based on the unsafe categories listed below.973974 Message:975 <message>${message}</message>976977 Unsafe Categories:978 <categories>979 ${unsafeCategoryStr}980 </categories>981982 Assign a risk level based on your confidence that the user's message should be moderated983 based on the defined unsafe categories:984985 0 - No risk986 1 - Low risk987 2 - Medium risk988 3 - High risk989990 Respond with ONLY a JSON object, using the format below:991 {992 "risk_level": <Numerical field denoting the risk level>,993 "categories": [Comma-separated list of violated categories],994 "explanation": <Optional. Only include if risk level is greater than 0>995 }996 Do not include markdown formatting or code fences in your response.`;997998 // Send the request to Claude for risk assessment999 const response = await client.messages.create({1000 model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs1001 max_tokens: 200,1002 messages: [{ role: "user", content: assessmentPrompt }]1003 });10041005 // Parse the JSON response from Claude1006 const textBlock = response.content.find((block) => block.type === "text");1007 if (!textBlock) {1008 throw new Error("Expected a text block in the response");1009 }1010 const assessment: RiskAssessment = JSON.parse(textBlock.text);10111012 // Extract the risk level, violated categories, and explanation from the assessment1013 const { risk_level: riskLevel, categories: violatedCategories, explanation } = assessment;10141015 return { riskLevel, violatedCategories, explanation };1016 }10171018 // Process each comment and print the results1019 for (const comment of userComments) {1020 console.log(`\nComment: ${comment}`);1021 const { riskLevel, violatedCategories, explanation } = await assessRiskLevel(1022 comment,1023 unsafeCategories1024 );10251026 console.log(`Risk Level: ${riskLevel}`);1027 if (violatedCategories.length > 0) {1028 console.log(`Violated Categories: ${violatedCategories.join(", ")}`);1029 }1030 if (explanation) {1031 console.log(`Explanation: ${explanation}`);1032 }1033 }1034 ```10351036 ```csharp C#1037 async Task<(int RiskLevel, List<string> ViolatedCategories, string? Explanation)> AssessRiskLevel(1038 string message,1039 IReadOnlyList<string> categories1040 )1041 {1042 // Convert the unsafe categories into a string, with each category on a new line1043 var unsafeCategoryText = string.Join("\n", categories);10441045 // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1046 var assessmentPrompt = $$"""10471048 Assess the risk level of the following message,1049 based on the unsafe categories listed below.10501051 Message:1052 <message>{{message}}</message>10531054 Unsafe Categories:1055 <categories>1056 {{unsafeCategoryText}}1057 </categories>10581059 Assign a risk level based on your confidence that the user's message should be moderated1060 based on the defined unsafe categories:10611062 0 - No risk1063 1 - Low risk1064 2 - Medium risk1065 3 - High risk10661067 Respond with ONLY a JSON object, using the format below:1068 {1069 "risk_level": <Numerical field denoting the risk level>,1070 "categories": [Comma-separated list of violated categories],1071 "explanation": <Optional. Only include if risk level is greater than 0>1072 }1073 Do not include markdown formatting or code fences in your response.1074 """;10751076 // Send the request to Claude for risk assessment1077 var response = await client.Messages.Create(1078 new()1079 {1080 Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1081 MaxTokens = 200,1082 Messages = [new() { Role = Role.User, Content = assessmentPrompt }],1083 }1084 );10851086 // Narrow the first content block to a text block, then parse Claude's JSON response1087 if (!response.Content[0].TryPickText(out var textBlock))1088 {1089 throw new InvalidOperationException("Expected a text response from Claude.");1090 }1091 var assessment = JsonNode.Parse(textBlock.Text)!;10921093 // Extract the risk level, violated categories, and explanation from the assessment1094 var riskLevel = assessment["risk_level"]!.GetValue<int>();1095 var violatedCategories = assessment["categories"]!1096 .AsArray()1097 .Select(category => category!.GetValue<string>())1098 .ToList();1099 var explanation = assessment["explanation"]?.GetValue<string>();11001101 return (riskLevel, violatedCategories, explanation);1102 }11031104 // Process each comment and print the results1105 foreach (var comment in userComments)1106 {1107 Console.WriteLine($"\nComment: {comment}");1108 var (riskLevel, violatedCategories, explanation) = await AssessRiskLevel(comment, unsafeCategories);11091110 Console.WriteLine($"Risk Level: {riskLevel}");1111 if (violatedCategories.Count > 0)1112 {1113 Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");1114 }1115 if (!string.IsNullOrEmpty(explanation))1116 {1117 Console.WriteLine($"Explanation: {explanation}");1118 }1119 }1120 ```11211122 ```go Go1123 func assessRiskLevel(message string, unsafeCategories []string) (int, []string, string) {1124 // Convert the unsafe categories into a string, with each category on a new line1125 unsafeCategoryStr := strings.Join(unsafeCategories, "\n")11261127 // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1128 assessmentPrompt := fmt.Sprintf(`1129 Assess the risk level of the following message,1130 based on the unsafe categories listed below.11311132 Message:1133 <message>%s</message>11341135 Unsafe Categories:1136 <categories>1137 %s1138 </categories>11391140 Assign a risk level based on your confidence that the user's message should be moderated1141 based on the defined unsafe categories:11421143 0 - No risk1144 1 - Low risk1145 2 - Medium risk1146 3 - High risk11471148 Respond with ONLY a JSON object, using the format below:1149 {1150 "risk_level": <Numerical field denoting the risk level>,1151 "categories": [Comma-separated list of violated categories],1152 "explanation": <Optional. Only include if risk level is greater than 0>1153 }1154 Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)11551156 // Send the request to Claude for risk assessment1157 response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{1158 Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1159 MaxTokens: 200,1160 Messages: []anthropic.MessageParam{1161 anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),1162 },1163 })1164 if err != nil {1165 log.Fatal(err)1166 }11671168 // Narrow the first content block to a text block before reading its text1169 textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)1170 if !ok {1171 log.Fatalf("expected a text block, got %q", response.Content[0].Type)1172 }11731174 // Parse the JSON response from Claude1175 var assessment struct {1176 RiskLevel int `json:"risk_level"`1177 Categories []string `json:"categories"`1178 Explanation string `json:"explanation"`1179 }1180 if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {1181 log.Fatal(err)1182 }11831184 // Return the risk level, violated categories, and explanation from the assessment1185 return assessment.RiskLevel, assessment.Categories, assessment.Explanation1186 }11871188 // assessAllRiskLevels processes each comment and prints the results.1189 func assessAllRiskLevels() {1190 for _, comment := range userComments {1191 fmt.Printf("\nComment: %s\n", comment)1192 riskLevel, violatedCategories, explanation := assessRiskLevel(comment, unsafeCategories)11931194 fmt.Printf("Risk Level: %d\n", riskLevel)1195 if len(violatedCategories) > 0 {1196 fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))1197 }1198 if explanation != "" {1199 fmt.Printf("Explanation: %s\n", explanation)1200 }1201 }1202 }12031204 ```12051206 ```java Java1207 record RiskAssessment(int riskLevel, List<String> violatedCategories, String explanation) {}12081209 RiskAssessment assessRiskLevel(String message, List<String> unsafeCategories)1210 throws JsonProcessingException {1211 // Convert the unsafe categories into a string, with each category on a new line1212 String unsafeCategoryStr = String.join("\n", unsafeCategories);12131214 // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1215 String assessmentPrompt = """12161217 Assess the risk level of the following message,1218 based on the unsafe categories listed below.12191220 Message:1221 <message>%s</message>12221223 Unsafe Categories:1224 <categories>1225 %s1226 </categories>12271228 Assign a risk level based on your confidence that the user's message should be moderated1229 based on the defined unsafe categories:12301231 0 - No risk1232 1 - Low risk1233 2 - Medium risk1234 3 - High risk12351236 Respond with ONLY a JSON object, using the format below:1237 {1238 "risk_level": <Numerical field denoting the risk level>,1239 "categories": [Comma-separated list of violated categories],1240 "explanation": <Optional. Only include if risk level is greater than 0>1241 }1242 Do not include markdown formatting or code fences in your response."""1243 .formatted(message, unsafeCategoryStr);12441245 // Send the request to Claude for risk assessment1246 Message response = client.messages().create(MessageCreateParams.builder()1247 .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs1248 .maxTokens(200)1249 .addUserMessage(assessmentPrompt)1250 .build());12511252 // Parse the JSON response from Claude1253 String assessmentJson = response.content().stream()1254 .flatMap(contentBlock -> contentBlock.text().stream())1255 .findFirst()1256 .orElseThrow()1257 .text();1258 ObjectMapper mapper = new ObjectMapper();1259 JsonNode assessment = mapper.readTree(assessmentJson);12601261 // Extract the risk level, violated categories, and explanation from the assessment1262 int riskLevel = assessment.required("risk_level").asInt();1263 JsonNode categoriesNode = assessment.required("categories");1264 List<String> violatedCategories = categoriesNode.isNull()1265 ? List.of()1266 : mapper.convertValue(categoriesNode, new TypeReference<List<String>>() {});1267 String explanation = assessment.hasNonNull("explanation")1268 ? assessment.get("explanation").asText()1269 : null;12701271 return new RiskAssessment(riskLevel, violatedCategories, explanation);1272 }12731274 // Process each comment and print the results1275 void printRiskLevels() throws JsonProcessingException {1276 for (String comment : userComments) {1277 IO.println("\nComment: " + comment);1278 RiskAssessment assessment = assessRiskLevel(comment, unsafeCategories);12791280 IO.println("Risk Level: " + assessment.riskLevel());1281 if (!assessment.violatedCategories().isEmpty()) {1282 IO.println("Violated Categories: " + String.join(", ", assessment.violatedCategories()));1283 }1284 if (assessment.explanation() != null && !assessment.explanation().isEmpty()) {1285 IO.println("Explanation: " + assessment.explanation());1286 }1287 }1288 }1289 ```12901291 ```php PHP1292 $assessRiskLevel = function (string $message, array $unsafeCategories) use ($client): array {1293 // Convert the unsafe categories into a string, with each category on a new line1294 $unsafeCategoryStr = implode("\n", $unsafeCategories);12951296 // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1297 $assessmentPrompt = <<<PROMPT12981299 Assess the risk level of the following message,1300 based on the unsafe categories listed below.13011302 Message:1303 <message>{$message}</message>13041305 Unsafe Categories:1306 <categories>1307 {$unsafeCategoryStr}1308 </categories>13091310 Assign a risk level based on your confidence that the user's message should be moderated1311 based on the defined unsafe categories:13121313 0 - No risk1314 1 - Low risk1315 2 - Medium risk1316 3 - High risk13171318 Respond with ONLY a JSON object, using the format below:1319 {1320 "risk_level": <Numerical field denoting the risk level>,1321 "categories": [Comma-separated list of violated categories],1322 "explanation": <Optional. Only include if risk level is greater than 0>1323 }1324 Do not include markdown formatting or code fences in your response.1325 PROMPT;13261327 // Send the request to Claude for risk assessment1328 $response = $client->messages->create(1329 model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs1330 maxTokens: 200,1331 messages: [['role' => 'user', 'content' => $assessmentPrompt]],1332 );13331334 // Parse the JSON response from Claude. The SDK decodes each content block1335 // into its concrete class, so find the TextBlock before reading the text.1336 $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)1337 ?? throw new RuntimeException('Expected a text block in the response.');1338 $assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);13391340 // Extract the risk level, violated categories, and explanation from the assessment1341 $riskLevel = $assessment['risk_level'];1342 $violatedCategories = $assessment['categories'];1343 $explanation = $assessment['explanation'] ?? null;13441345 return [$riskLevel, $violatedCategories, $explanation];1346 };13471348 // Process each comment and print the results1349 foreach ($userComments as $comment) {1350 echo "\nComment: {$comment}\n";1351 [$riskLevel, $violatedCategories, $explanation] = $assessRiskLevel($comment, $unsafeCategories);13521353 echo "Risk Level: {$riskLevel}\n";1354 if ($violatedCategories) {1355 echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";1356 }1357 if ($explanation) {1358 echo "Explanation: {$explanation}\n";1359 }1360 }1361 ```13621363 ```ruby Ruby1364 def assess_risk_level(message, unsafe_categories)1365 # Convert the unsafe categories into a string, with each category on a new line1366 unsafe_category_str = unsafe_categories.join("\n")13671368 # Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1369 assessment_prompt = <<~PROMPT.chomp13701371 Assess the risk level of the following message,1372 based on the unsafe categories listed below.13731374 Message:1375 <message>#{message}</message>13761377 Unsafe Categories:1378 <categories>1379 #{unsafe_category_str}1380 </categories>13811382 Assign a risk level based on your confidence that the user's message should be moderated1383 based on the defined unsafe categories:13841385 0 - No risk1386 1 - Low risk1387 2 - Medium risk1388 3 - High risk13891390 Respond with ONLY a JSON object, using the format below:1391 {1392 "risk_level": <Numerical field denoting the risk level>,1393 "categories": [Comma-separated list of violated categories],1394 "explanation": <Optional. Only include if risk level is greater than 0>1395 }1396 Do not include markdown formatting or code fences in your response.1397 PROMPT13981399 # Send the request to Claude for risk assessment1400 response = CLIENT.messages.create(1401 model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs1402 max_tokens: 200,1403 messages: [{role: :user, content: assessment_prompt}]1404 )14051406 # Parse the JSON response from Claude1407 text_block = response.content.find { it.type == :text }1408 assessment = JSON.parse(text_block.text)14091410 # Extract the risk level, violated categories, and explanation from the assessment1411 risk_level = assessment["risk_level"]1412 violated_categories = assessment["categories"]1413 explanation = assessment["explanation"]14141415 [risk_level, violated_categories, explanation]1416 end141714181419 # Process each comment and print the results1420 USER_COMMENTS.each do |comment|1421 puts "\nComment: #{comment}"1422 risk_level, violated_categories, explanation = assess_risk_level(comment, UNSAFE_CATEGORIES)14231424 puts "Risk Level: #{risk_level}"1425 puts "Violated Categories: #{violated_categories.join(", ")}" if violated_categories&.any?1426 puts "Explanation: #{explanation}" if explanation1427 end1428 ```1429</CodeGroup>14301431This code implements an `assess_risk_level` function that uses Claude to evaluate the risk level of a message. The function accepts a message and the unsafe categories as inputs.14321433Within the function, a prompt is generated for Claude, including the message to be assessed, the unsafe categories, and specific instructions for evaluating the risk level. The prompt instructs Claude to respond with a JSON object that includes the risk level, the violated categories, and an optional explanation.14341435This approach enables flexible content moderation by assigning risk levels. It can be seamlessly integrated into a larger system to automate content filtering or flag comments for human review based on their assessed risk level. For instance, when running this code, the comment `Delete this post now or you better hide. I am coming after you and your family.` is identified as high risk because of its dangerous threat. Conversely, the comment `Stay away from the 5G cellphones!! They are using 5G to control you.` is categorized as medium risk.14361437### Deploy your prompt14381439Once you are confident in the quality of your solution, it's time to deploy it to production. Here are some best practices to follow when using content moderation in production:144014411. **Provide clear feedback to users:** When user input is blocked or a response is flagged because of content moderation, provide informative and constructive feedback to help users understand why their message was flagged and how they can rephrase it appropriately. In the earlier coding examples, this is done through the `explanation` field in the Claude response.144214432. **Analyze moderated content:** Keep track of the types of content being flagged by your moderation system to identify trends and potential areas for improvement.144414453. **Continuously evaluate and improve:** Regularly assess the performance of your content moderation system using metrics such as precision and recall tracking. Use this data to iteratively refine your moderation prompts, keywords, and assessment criteria.14461447***14481449## Improve performance14501451In complex scenarios, it may be helpful to consider additional strategies to improve performance beyond standard [prompt engineering techniques](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview). Here are some advanced strategies:14521453### Define topics and provide examples14541455In addition to listing the unsafe categories in the prompt, further improvements can be made by providing definitions and phrases related to each category.14561457<CodeGroup exclude="shell">1458 ```python Python1459 # Categories considered unsafe for content moderation, with their definitions1460 unsafe_category_definitions = {1461 "Child Exploitation": "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",1462 "Conspiracy Theories": """Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1463 which can incite harm or undermine public trust in institutions or facts.""",1464 "Hate": """Content that is hateful toward people on the basis of their protected characteristics1465 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1466 or content that perpetuates negative stereotypes.""",1467 "Indiscriminate Weapons": "Content that enables the creation of weapons of mass destruction.",1468 "Intellectual Property": "Content that violates the intellectual property rights of third parties.",1469 "Non-Violent Crimes": "Content that enables, encourages, or excuses the commission of non-violent crimes.",1470 "Privacy": "Content that contains sensitive, personal information about private individuals.",1471 "Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",1472 "Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",1473 "Sexual Content": "Content that contains erotic descriptions or explicit depictions of sex acts.",1474 "Specialized Advice": """Content that contains financial, medical, or legal advice.1475 Financial advice includes guidance on investments, stocks, bonds, or any financial planning.""",1476 "Violent Crimes": "Content that enables, encourages, or excuses the commission of violent crimes.",1477 }147814791480 def moderate_message_with_definitions(message, unsafe_category_definitions):1481 # Format unsafe categories string, with each category and its definition on a new line1482 unsafe_category_str = "\n".join(1483 f"{category}: {definition}"1484 for category, definition in unsafe_category_definitions.items()1485 )14861487 # Construct the prompt for Claude, including the message and unsafe categories1488 assessment_prompt = f"""Determine whether the following message warrants moderation, based on the unsafe categories outlined below.14891490 Message:1491 <message>{message}</message>14921493 Unsafe Categories and Their Definitions:1494 <categories>1495 {unsafe_category_str}1496 </categories>14971498 It's important that you remember all unsafe categories and their definitions.14991500 Respond with ONLY a JSON object, using the format below:1501 {{1502 "violation": <Boolean field denoting whether the message should be moderated>,1503 "categories": [Comma-separated list of violated categories],1504 "explanation": [Optional. Only include if there is a violation.]1505 }}1506 Do not include markdown formatting or code fences in your response."""15071508 # Send the request to Claude for content moderation1509 response = client.messages.create(1510 model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs1511 max_tokens=200,1512 messages=[{"role": "user", "content": assessment_prompt}],1513 )15141515 # Parse the JSON response from Claude1516 text_block = next(block for block in response.content if block.type == "text")1517 assessment = json.loads(text_block.text)15181519 # Extract the violation status from the assessment1520 contains_violation = assessment["violation"]15211522 # If there's a violation, get the categories and explanation; otherwise, use empty defaults1523 violated_categories = assessment.get("categories", []) if contains_violation else []1524 explanation = assessment.get("explanation") if contains_violation else None15251526 return contains_violation, violated_categories, explanation152715281529 # Process each comment and print the results1530 for comment in user_comments:1531 print(f"\nComment: {comment}")1532 violation, violated_categories, explanation = moderate_message_with_definitions(1533 comment, unsafe_category_definitions1534 )15351536 if violation:1537 print(f"Violated Categories: {', '.join(violated_categories)}")1538 print(f"Explanation: {explanation}")1539 else:1540 print("No issues detected.")1541 ```15421543 ```typescript TypeScript1544 // Shape of the JSON assessment Claude returns1545 interface DefinitionBasedAssessment {1546 violation: boolean;1547 categories?: string[];1548 explanation?: string;1549 }15501551 // Categories considered unsafe for content moderation, with their definitions1552 // (object keys preserve insertion order, so categories render in this order)1553 const unsafeCategoryDefinitions: Record<string, string> = {1554 "Child Exploitation":1555 "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",1556 "Conspiracy Theories": `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1557 which can incite harm or undermine public trust in institutions or facts.`,1558 "Hate": `Content that is hateful toward people on the basis of their protected characteristics1559 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1560 or content that perpetuates negative stereotypes.`,1561 "Indiscriminate Weapons":1562 "Content that enables the creation of weapons of mass destruction.",1563 "Intellectual Property":1564 "Content that violates the intellectual property rights of third parties.",1565 "Non-Violent Crimes":1566 "Content that enables, encourages, or excuses the commission of non-violent crimes.",1567 "Privacy":1568 "Content that contains sensitive, personal information about private individuals.",1569 "Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",1570 "Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",1571 "Sexual Content":1572 "Content that contains erotic descriptions or explicit depictions of sex acts.",1573 "Specialized Advice": `Content that contains financial, medical, or legal advice.1574 Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`,1575 "Violent Crimes":1576 "Content that enables, encourages, or excuses the commission of violent crimes."1577 };15781579 async function moderateMessageWithDefinitions(1580 message: string,1581 unsafeCategoryDefinitions: Record<string, string>1582 ): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {1583 // Format the unsafe categories string, with each category and its definition on a new line1584 const unsafeCategoryStr = Object.entries(unsafeCategoryDefinitions)1585 .map(([category, definition]) => `${category}: ${definition}`)1586 .join("\n");15871588 // Construct the prompt for Claude, including the message and unsafe categories1589 const assessmentPrompt = `Determine whether the following message warrants moderation, based on the unsafe categories outlined below.15901591 Message:1592 <message>${message}</message>15931594 Unsafe Categories and Their Definitions:1595 <categories>1596 ${unsafeCategoryStr}1597 </categories>15981599 It's important that you remember all unsafe categories and their definitions.16001601 Respond with ONLY a JSON object, using the format below:1602 {1603 "violation": <Boolean field denoting whether the message should be moderated>,1604 "categories": [Comma-separated list of violated categories],1605 "explanation": [Optional. Only include if there is a violation.]1606 }1607 Do not include markdown formatting or code fences in your response.`;16081609 // Send the request to Claude for content moderation1610 const response = await client.messages.create({1611 model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs1612 max_tokens: 200,1613 messages: [{ role: "user", content: assessmentPrompt }]1614 });16151616 // Parse the JSON response from Claude1617 const textBlock = response.content.find((block) => block.type === "text");1618 if (!textBlock) {1619 throw new Error("Expected a text block in the response");1620 }1621 const assessment: DefinitionBasedAssessment = JSON.parse(textBlock.text);16221623 // Extract the violation status from the assessment1624 const containsViolation = assessment.violation;16251626 // If there's a violation, get the categories and explanation; otherwise, use empty defaults1627 const violatedCategories = containsViolation ? assessment.categories ?? [] : [];1628 const explanation = containsViolation ? assessment.explanation : undefined;16291630 return { violation: containsViolation, violatedCategories, explanation };1631 }16321633 // Process each comment and print the results1634 for (const comment of userComments) {1635 console.log(`\nComment: ${comment}`);1636 const { violation, violatedCategories, explanation } = await moderateMessageWithDefinitions(1637 comment,1638 unsafeCategoryDefinitions1639 );16401641 if (violation) {1642 console.log(`Violated Categories: ${violatedCategories.join(", ")}`);1643 console.log(`Explanation: ${explanation}`);1644 } else {1645 console.log("No issues detected.");1646 }1647 }1648 ```16491650 ```csharp C#1651 // Categories considered unsafe for content moderation, with their definitions.1652 // The entries stay in insertion order, so the rendered prompt lists categories1653 // in exactly this order.1654 (string Category, string Definition)[] unsafeCategoryDefinitions =1655 [1656 (1657 "Child Exploitation",1658 "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."1659 ),1660 (1661 "Conspiracy Theories",1662 """1663 Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1664 which can incite harm or undermine public trust in institutions or facts.1665 """1666 ),1667 (1668 "Hate",1669 """1670 Content that is hateful toward people on the basis of their protected characteristics1671 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1672 or content that perpetuates negative stereotypes.1673 """1674 ),1675 ("Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."),1676 ("Intellectual Property", "Content that violates the intellectual property rights of third parties."),1677 ("Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."),1678 ("Privacy", "Content that contains sensitive, personal information about private individuals."),1679 ("Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."),1680 ("Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."),1681 ("Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."),1682 (1683 "Specialized Advice",1684 """1685 Content that contains financial, medical, or legal advice.1686 Financial advice includes guidance on investments, stocks, bonds, or any financial planning.1687 """1688 ),1689 ("Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."),1690 ];169116921693 async Task<(bool ContainsViolation, List<string> ViolatedCategories, string? Explanation)> ModerateMessageWithDefinitions(1694 string message,1695 IReadOnlyList<(string Category, string Definition)> categoryDefinitions1696 )1697 {1698 // Format the unsafe categories string, with each category and its definition on a new line1699 var unsafeCategoryText = string.Join(1700 "\n",1701 categoryDefinitions.Select(entry => $"{entry.Category}: {entry.Definition}")1702 );17031704 // Construct the prompt for Claude, including the message and unsafe categories1705 var assessmentPrompt = $$"""1706 Determine whether the following message warrants moderation, based on the unsafe categories outlined below.17071708 Message:1709 <message>{{message}}</message>17101711 Unsafe Categories and Their Definitions:1712 <categories>1713 {{unsafeCategoryText}}1714 </categories>17151716 It's important that you remember all unsafe categories and their definitions.17171718 Respond with ONLY a JSON object, using the format below:1719 {1720 "violation": <Boolean field denoting whether the message should be moderated>,1721 "categories": [Comma-separated list of violated categories],1722 "explanation": [Optional. Only include if there is a violation.]1723 }1724 Do not include markdown formatting or code fences in your response.1725 """;17261727 // Send the request to Claude for content moderation1728 var response = await client.Messages.Create(1729 new()1730 {1731 Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1732 MaxTokens = 200,1733 Messages = [new() { Role = Role.User, Content = assessmentPrompt }],1734 }1735 );17361737 // Narrow the first content block to a text block, then parse Claude's JSON response1738 if (!response.Content[0].TryPickText(out var textBlock))1739 {1740 throw new InvalidOperationException("Expected a text response from Claude.");1741 }1742 var assessment = JsonNode.Parse(textBlock.Text)!;17431744 // Extract the violation status from the assessment1745 var containsViolation = assessment["violation"]!.GetValue<bool>();17461747 // If there's a violation, get the categories and explanation; otherwise, use empty defaults1748 List<string> violatedCategories = containsViolation1749 ? assessment["categories"]?.AsArray().Select(category => category!.GetValue<string>()).ToList() ?? []1750 : [];1751 var explanation = containsViolation ? assessment["explanation"]?.GetValue<string>() : null;17521753 return (containsViolation, violatedCategories, explanation);1754 }17551756 // Process each comment and print the results1757 foreach (var comment in userComments)1758 {1759 Console.WriteLine($"\nComment: {comment}");1760 var (violation, violatedCategories, explanation) = await ModerateMessageWithDefinitions(1761 comment,1762 unsafeCategoryDefinitions1763 );17641765 if (violation)1766 {1767 Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");1768 Console.WriteLine($"Explanation: {explanation}");1769 }1770 else1771 {1772 Console.WriteLine("No issues detected.");1773 }1774 }1775 ```17761777 ```go Go1778 // Categories considered unsafe for content moderation, with their definitions.1779 // A slice of category/definition pairs (rather than a map) keeps the rendered1780 // order stable; Go maps iterate in random order.1781 type categoryDefinition struct {1782 category string1783 definition string1784 }17851786 var unsafeCategoryDefinitions = []categoryDefinition{1787 {"Child Exploitation", "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."},1788 {"Conspiracy Theories", `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1789 which can incite harm or undermine public trust in institutions or facts.`},1790 {"Hate", `Content that is hateful toward people on the basis of their protected characteristics1791 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1792 or content that perpetuates negative stereotypes.`},1793 {"Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."},1794 {"Intellectual Property", "Content that violates the intellectual property rights of third parties."},1795 {"Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."},1796 {"Privacy", "Content that contains sensitive, personal information about private individuals."},1797 {"Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."},1798 {"Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."},1799 {"Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."},1800 {"Specialized Advice", `Content that contains financial, medical, or legal advice.1801 Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`},1802 {"Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."},1803 }18041805 func moderateMessageWithDefinitions(message string, unsafeCategoryDefinitions []categoryDefinition) (bool, []string, string) {1806 // Format unsafe categories string, with each category and its definition on a new line1807 categoryLines := make([]string, len(unsafeCategoryDefinitions))1808 for i, entry := range unsafeCategoryDefinitions {1809 categoryLines[i] = fmt.Sprintf("%s: %s", entry.category, entry.definition)1810 }1811 unsafeCategoryStr := strings.Join(categoryLines, "\n")18121813 // Construct the prompt for Claude, including the message and unsafe categories1814 assessmentPrompt := fmt.Sprintf(`Determine whether the following message warrants moderation, based on the unsafe categories outlined below.18151816 Message:1817 <message>%s</message>18181819 Unsafe Categories and Their Definitions:1820 <categories>1821 %s1822 </categories>18231824 It's important that you remember all unsafe categories and their definitions.18251826 Respond with ONLY a JSON object, using the format below:1827 {1828 "violation": <Boolean field denoting whether the message should be moderated>,1829 "categories": [Comma-separated list of violated categories],1830 "explanation": [Optional. Only include if there is a violation.]1831 }1832 Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)18331834 // Send the request to Claude for content moderation1835 response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{1836 Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1837 MaxTokens: 200,1838 Messages: []anthropic.MessageParam{1839 anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),1840 },1841 })1842 if err != nil {1843 log.Fatal(err)1844 }18451846 // Narrow the first content block to a text block before reading its text1847 textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)1848 if !ok {1849 log.Fatalf("expected a text block, got %q", response.Content[0].Type)1850 }18511852 // Parse the JSON response from Claude1853 var assessment struct {1854 Violation bool `json:"violation"`1855 Categories []string `json:"categories"`1856 Explanation string `json:"explanation"`1857 }1858 if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {1859 log.Fatal(err)1860 }18611862 // If there's a violation, return the categories and explanation; otherwise, use empty defaults1863 if !assessment.Violation {1864 return false, nil, ""1865 }1866 return true, assessment.Categories, assessment.Explanation1867 }18681869 // moderateAllCommentsWithDefinitions processes each comment and prints the results.1870 func moderateAllCommentsWithDefinitions() {1871 for _, comment := range userComments {1872 fmt.Printf("\nComment: %s\n", comment)1873 violation, violatedCategories, explanation := moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions)18741875 if violation {1876 fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))1877 fmt.Printf("Explanation: %s\n", explanation)1878 } else {1879 fmt.Println("No issues detected.")1880 }1881 }1882 }18831884 ```18851886 ```java Java1887 // Categories considered unsafe for content moderation, with their definitions1888 record CategoryDefinition(String category, String definition) {}18891890 final List<CategoryDefinition> unsafeCategoryDefinitions = List.of(1891 new CategoryDefinition(1892 "Child Exploitation",1893 "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."),1894 new CategoryDefinition(1895 "Conspiracy Theories",1896 """1897 Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1898 which can incite harm or undermine public trust in institutions or facts."""),1899 new CategoryDefinition(1900 "Hate",1901 """1902 Content that is hateful toward people on the basis of their protected characteristics1903 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1904 or content that perpetuates negative stereotypes."""),1905 new CategoryDefinition(1906 "Indiscriminate Weapons",1907 "Content that enables the creation of weapons of mass destruction."),1908 new CategoryDefinition(1909 "Intellectual Property",1910 "Content that violates the intellectual property rights of third parties."),1911 new CategoryDefinition(1912 "Non-Violent Crimes",1913 "Content that enables, encourages, or excuses the commission of non-violent crimes."),1914 new CategoryDefinition(1915 "Privacy",1916 "Content that contains sensitive, personal information about private individuals."),1917 new CategoryDefinition(1918 "Self-Harm",1919 "Content that enables, encourages, or excuses acts of intentional self-harm."),1920 new CategoryDefinition(1921 "Sex Crimes",1922 "Content that enables, encourages, or excuses the commission of sex crimes."),1923 new CategoryDefinition(1924 "Sexual Content",1925 "Content that contains erotic descriptions or explicit depictions of sex acts."),1926 new CategoryDefinition(1927 "Specialized Advice",1928 """1929 Content that contains financial, medical, or legal advice.1930 Financial advice includes guidance on investments, stocks, bonds, or any financial planning."""),1931 new CategoryDefinition(1932 "Violent Crimes",1933 "Content that enables, encourages, or excuses the commission of violent crimes."));19341935 record ModerationDecision(boolean violation, List<String> violatedCategories, String explanation) {}19361937 ModerationDecision moderateMessageWithDefinitions(1938 String message, List<CategoryDefinition> unsafeCategoryDefinitions)1939 throws JsonProcessingException {1940 // Format unsafe categories string, with each category and its definition on a new line1941 String unsafeCategoryStr = unsafeCategoryDefinitions.stream()1942 .map(categoryDefinition ->1943 categoryDefinition.category() + ": " + categoryDefinition.definition())1944 .collect(Collectors.joining("\n"));19451946 // Construct the prompt for Claude, including the message and unsafe categories1947 String assessmentPrompt = """1948 Determine whether the following message warrants moderation, based on the unsafe categories outlined below.19491950 Message:1951 <message>%s</message>19521953 Unsafe Categories and Their Definitions:1954 <categories>1955 %s1956 </categories>19571958 It's important that you remember all unsafe categories and their definitions.19591960 Respond with ONLY a JSON object, using the format below:1961 {1962 "violation": <Boolean field denoting whether the message should be moderated>,1963 "categories": [Comma-separated list of violated categories],1964 "explanation": [Optional. Only include if there is a violation.]1965 }1966 Do not include markdown formatting or code fences in your response."""1967 .formatted(message, unsafeCategoryStr);19681969 // Send the request to Claude for content moderation1970 Message response = client.messages().create(MessageCreateParams.builder()1971 .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs1972 .maxTokens(200)1973 .addUserMessage(assessmentPrompt)1974 .build());19751976 // Parse the JSON response from Claude1977 String assessmentJson = response.content().stream()1978 .flatMap(contentBlock -> contentBlock.text().stream())1979 .findFirst()1980 .orElseThrow()1981 .text();1982 ObjectMapper mapper = new ObjectMapper();1983 JsonNode assessment = mapper.readTree(assessmentJson);19841985 // Extract the violation status from the assessment1986 boolean containsViolation = assessment.required("violation").asBoolean();19871988 // If there's a violation, get the categories and explanation; otherwise, use empty defaults1989 List<String> violatedCategories = containsViolation && assessment.has("categories")1990 ? mapper.convertValue(assessment.get("categories"), new TypeReference<List<String>>() {})1991 : List.of();1992 String explanation = containsViolation && assessment.hasNonNull("explanation")1993 ? assessment.get("explanation").asText()1994 : null;19951996 return new ModerationDecision(containsViolation, violatedCategories, explanation);1997 }19981999 // Process each comment and print the results2000 void printModerationResultsWithDefinitions() throws JsonProcessingException {2001 for (String comment : userComments) {2002 IO.println("\nComment: " + comment);2003 ModerationDecision result = moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions);20042005 if (result.violation()) {2006 IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));2007 IO.println("Explanation: " + result.explanation());2008 } else {2009 IO.println("No issues detected.");2010 }2011 }2012 }2013 ```20142015 ```php PHP2016 // Categories considered unsafe for content moderation, with their definitions2017 $unsafeCategoryDefinitions = [2018 'Child Exploitation' => 'Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.',2019 'Conspiracy Theories' => 'Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,2020 which can incite harm or undermine public trust in institutions or facts.',2021 'Hate' => 'Content that is hateful toward people on the basis of their protected characteristics2022 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),2023 or content that perpetuates negative stereotypes.',2024 'Indiscriminate Weapons' => 'Content that enables the creation of weapons of mass destruction.',2025 'Intellectual Property' => 'Content that violates the intellectual property rights of third parties.',2026 'Non-Violent Crimes' => 'Content that enables, encourages, or excuses the commission of non-violent crimes.',2027 'Privacy' => 'Content that contains sensitive, personal information about private individuals.',2028 'Self-Harm' => 'Content that enables, encourages, or excuses acts of intentional self-harm.',2029 'Sex Crimes' => 'Content that enables, encourages, or excuses the commission of sex crimes.',2030 'Sexual Content' => 'Content that contains erotic descriptions or explicit depictions of sex acts.',2031 'Specialized Advice' => 'Content that contains financial, medical, or legal advice.2032 Financial advice includes guidance on investments, stocks, bonds, or any financial planning.',2033 'Violent Crimes' => 'Content that enables, encourages, or excuses the commission of violent crimes.',2034 ];20352036 $moderateMessageWithDefinitions = function (string $message, array $unsafeCategoryDefinitions) use ($client): array {2037 // Format the unsafe categories string, with each category and its definition on a new line2038 $categoryLines = [];2039 foreach ($unsafeCategoryDefinitions as $category => $definition) {2040 $categoryLines[] = "{$category}: {$definition}";2041 }2042 $unsafeCategoryStr = implode("\n", $categoryLines);20432044 // Construct the prompt for Claude, including the message and unsafe categories2045 $assessmentPrompt = <<<PROMPT2046 Determine whether the following message warrants moderation, based on the unsafe categories outlined below.20472048 Message:2049 <message>{$message}</message>20502051 Unsafe Categories and Their Definitions:2052 <categories>2053 {$unsafeCategoryStr}2054 </categories>20552056 It's important that you remember all unsafe categories and their definitions.20572058 Respond with ONLY a JSON object, using the format below:2059 {2060 "violation": <Boolean field denoting whether the message should be moderated>,2061 "categories": [Comma-separated list of violated categories],2062 "explanation": [Optional. Only include if there is a violation.]2063 }2064 Do not include markdown formatting or code fences in your response.2065 PROMPT;20662067 // Send the request to Claude for content moderation2068 $response = $client->messages->create(2069 model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs2070 maxTokens: 200,2071 messages: [['role' => 'user', 'content' => $assessmentPrompt]],2072 );20732074 // Parse the JSON response from Claude. The SDK decodes each content block2075 // into its concrete class, so find the TextBlock before reading the text.2076 $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)2077 ?? throw new RuntimeException('Expected a text block in the response.');2078 $assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);20792080 // Extract the violation status from the assessment2081 $containsViolation = $assessment['violation'];20822083 // If there's a violation, get the categories and explanation; otherwise, use empty defaults2084 $violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];2085 $explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;20862087 return [$containsViolation, $violatedCategories, $explanation];2088 };20892090 // Process each comment and print the results2091 foreach ($userComments as $comment) {2092 echo "\nComment: {$comment}\n";2093 [$violation, $violatedCategories, $explanation] = $moderateMessageWithDefinitions($comment, $unsafeCategoryDefinitions);20942095 if ($violation) {2096 echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";2097 echo "Explanation: {$explanation}\n";2098 } else {2099 echo "No issues detected.\n";2100 }2101 }2102 ```21032104 ```ruby Ruby2105 # Categories considered unsafe for content moderation, with their definitions2106 UNSAFE_CATEGORY_DEFINITIONS = {2107 "Child Exploitation" => "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",2108 "Conspiracy Theories" => "Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,2109 which can incite harm or undermine public trust in institutions or facts.",2110 "Hate" => "Content that is hateful toward people on the basis of their protected characteristics2111 (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),2112 or content that perpetuates negative stereotypes.",2113 "Indiscriminate Weapons" => "Content that enables the creation of weapons of mass destruction.",2114 "Intellectual Property" => "Content that violates the intellectual property rights of third parties.",2115 "Non-Violent Crimes" => "Content that enables, encourages, or excuses the commission of non-violent crimes.",2116 "Privacy" => "Content that contains sensitive, personal information about private individuals.",2117 "Self-Harm" => "Content that enables, encourages, or excuses acts of intentional self-harm.",2118 "Sex Crimes" => "Content that enables, encourages, or excuses the commission of sex crimes.",2119 "Sexual Content" => "Content that contains erotic descriptions or explicit depictions of sex acts.",2120 "Specialized Advice" => "Content that contains financial, medical, or legal advice.2121 Financial advice includes guidance on investments, stocks, bonds, or any financial planning.",2122 "Violent Crimes" => "Content that enables, encourages, or excuses the commission of violent crimes."2123 }212421252126 def moderate_message_with_definitions(message, unsafe_category_definitions)2127 # Format the unsafe categories string, with each category and its definition on a new line2128 unsafe_category_str = unsafe_category_definitions2129 .map { |category, definition| "#{category}: #{definition}" }2130 .join("\n")21312132 # Construct the prompt for Claude, including the message and unsafe categories2133 assessment_prompt = <<~PROMPT.chomp2134 Determine whether the following message warrants moderation, based on the unsafe categories outlined below.21352136 Message:2137 <message>#{message}</message>21382139 Unsafe Categories and Their Definitions:2140 <categories>2141 #{unsafe_category_str}2142 </categories>21432144 It's important that you remember all unsafe categories and their definitions.21452146 Respond with ONLY a JSON object, using the format below:2147 {2148 "violation": <Boolean field denoting whether the message should be moderated>,2149 "categories": [Comma-separated list of violated categories],2150 "explanation": [Optional. Only include if there is a violation.]2151 }2152 Do not include markdown formatting or code fences in your response.2153 PROMPT21542155 # Send the request to Claude for content moderation2156 response = CLIENT.messages.create(2157 model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs2158 max_tokens: 200,2159 messages: [{role: :user, content: assessment_prompt}]2160 )21612162 # Parse the JSON response from Claude2163 text_block = response.content.find { it.type == :text }2164 assessment = JSON.parse(text_block.text)21652166 # Extract the violation status from the assessment2167 contains_violation = assessment["violation"]21682169 # If there's a violation, get the categories and explanation; otherwise, use empty defaults2170 violated_categories = contains_violation ? assessment.fetch("categories", []) : []2171 explanation = contains_violation ? assessment["explanation"] : nil21722173 [contains_violation, violated_categories, explanation]2174 end217521762177 # Process each comment and print the results2178 USER_COMMENTS.each do |comment|2179 puts "\nComment: #{comment}"2180 violation, violated_categories, explanation = moderate_message_with_definitions(comment, UNSAFE_CATEGORY_DEFINITIONS)21812182 if violation2183 puts "Violated Categories: #{violated_categories.join(", ")}"2184 puts "Explanation: #{explanation}"2185 else2186 puts "No issues detected."2187 end2188 end2189 ```2190</CodeGroup>21912192The `moderate_message_with_definitions` function expands upon the earlier `moderate_message` function by allowing each unsafe category to be paired with a detailed definition. This occurs in the code by replacing the `unsafe_categories` collection from the original function with an `unsafe_category_definitions` mapping. This mapping pairs each unsafe category with its corresponding definition. Both the category names and their definitions are included in the prompt.21932194Notably, the definition for the `Specialized Advice` category now specifies the types of financial advice that should be prohibited. As a result, the comment `It is a great time to invest in gold!`, which previously passed the `moderate_message` assessment, now triggers a violation.21952196### Consider batch processing21972198To reduce costs in situations where real-time moderation isn't necessary, consider moderating messages in batches. Include multiple messages within the prompt's context, and ask Claude to assess which messages should be moderated.21992200<CodeGroup exclude="shell">2201 ```python Python2202 def batch_moderate_messages(messages, unsafe_categories):2203 # Convert the unsafe categories into a string, with each category on a new line2204 unsafe_category_str = "\n".join(unsafe_categories)22052206 # Format messages string, with each message wrapped in XML-like tags and given an ID2207 messages_str = "\n".join(2208 [f"<message id={idx}>{msg}</message>" for idx, msg in enumerate(messages)]2209 )22102211 # Construct the prompt for Claude, including the messages and unsafe categories2212 assessment_prompt = f"""Determine the messages to moderate, based on the unsafe categories outlined below.22132214 Messages:2215 <messages>2216 {messages_str}2217 </messages>22182219 Unsafe Categories:2220 <categories>2221 {unsafe_category_str}2222 </categories>22232224 Respond with ONLY a JSON object, using the format below:2225 {{2226 "violations": [2227 {{2228 "id": <message id>,2229 "categories": [list of violated categories],2230 "explanation": <Explanation of why there's a violation>2231 }}2232 ]2233 }}22342235 Important Notes:2236 - Remember to analyze every message for a violation.2237 - Select any number of violations that reasonably apply.2238 - Do not include markdown formatting or code fences in your response."""22392240 # Send the request to Claude for content moderation2241 response = client.messages.create(2242 model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs2243 max_tokens=2048, # Increased max token count to handle batches2244 messages=[{"role": "user", "content": assessment_prompt}],2245 )22462247 # Parse the JSON response from Claude2248 text_block = next(block for block in response.content if block.type == "text")2249 assessment = json.loads(text_block.text)2250 return assessment225122522253 # Process the batch of comments and get the response2254 response_obj = batch_moderate_messages(user_comments, unsafe_categories)22552256 # Print the results for each detected violation2257 for violation in response_obj["violations"]:2258 print(f"""Comment: {user_comments[violation["id"]]}2259 Violated Categories: {", ".join(violation["categories"])}2260 Explanation: {violation["explanation"]}2261 """)2262 ```22632264 ```typescript TypeScript2265 // Shape of the JSON batch assessment Claude returns2266 interface BatchAssessment {2267 violations: {2268 id: number;2269 categories: string[];2270 explanation: string;2271 }[];2272 }22732274 async function batchModerateMessages(2275 messages: string[],2276 unsafeCategories: string[]2277 ): Promise<BatchAssessment> {2278 // Convert the unsafe categories into a string, with each category on a new line2279 const unsafeCategoryStr = unsafeCategories.join("\n");22802281 // Format the messages string, with each message wrapped in XML-like tags and given an ID2282 const messagesStr = messages2283 .map((msg, idx) => `<message id=${idx}>${msg}</message>`)2284 .join("\n");22852286 // Construct the prompt for Claude, including the messages and unsafe categories2287 const assessmentPrompt = `Determine the messages to moderate, based on the unsafe categories outlined below.22882289 Messages:2290 <messages>2291 ${messagesStr}2292 </messages>22932294 Unsafe Categories:2295 <categories>2296 ${unsafeCategoryStr}2297 </categories>22982299 Respond with ONLY a JSON object, using the format below:2300 {2301 "violations": [2302 {2303 "id": <message id>,2304 "categories": [list of violated categories],2305 "explanation": <Explanation of why there's a violation>2306 }2307 ]2308 }23092310 Important Notes:2311 - Remember to analyze every message for a violation.2312 - Select any number of violations that reasonably apply.2313 - Do not include markdown formatting or code fences in your response.`;23142315 // Send the request to Claude for content moderation2316 const response = await client.messages.create({2317 model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs2318 max_tokens: 2048, // Increased max token count to handle batches2319 messages: [{ role: "user", content: assessmentPrompt }]2320 });23212322 // Parse the JSON response from Claude2323 const textBlock = response.content.find((block) => block.type === "text");2324 if (!textBlock) {2325 throw new Error("Expected a text block in the response");2326 }2327 const assessment: BatchAssessment = JSON.parse(textBlock.text);2328 return assessment;2329 }23302331 // Process the batch of comments and get the response2332 const batchAssessment = await batchModerateMessages(userComments, unsafeCategories);23332334 // Print the results for each detected violation2335 for (const violation of batchAssessment.violations) {2336 console.log(`Comment: ${userComments[violation.id]}2337 Violated Categories: ${violation.categories.join(", ")}2338 Explanation: ${violation.explanation}2339 `);2340 }2341 ```23422343 ```csharp C#2344 async Task<JsonNode> BatchModerateMessages(IReadOnlyList<string> messages, IReadOnlyList<string> categories)2345 {2346 // Convert the unsafe categories into a string, with each category on a new line2347 var unsafeCategoryText = string.Join("\n", categories);23482349 // Format the messages string, with each message wrapped in XML-like tags and given an ID2350 var messagesText = string.Join(2351 "\n",2352 messages.Select((message, index) => $"<message id={index}>{message}</message>")2353 );23542355 // Construct the prompt for Claude, including the messages and unsafe categories2356 var assessmentPrompt = $$"""2357 Determine the messages to moderate, based on the unsafe categories outlined below.23582359 Messages:2360 <messages>2361 {{messagesText}}2362 </messages>23632364 Unsafe Categories:2365 <categories>2366 {{unsafeCategoryText}}2367 </categories>23682369 Respond with ONLY a JSON object, using the format below:2370 {2371 "violations": [2372 {2373 "id": <message id>,2374 "categories": [list of violated categories],2375 "explanation": <Explanation of why there's a violation>2376 }2377 ]2378 }23792380 Important Notes:2381 - Remember to analyze every message for a violation.2382 - Select any number of violations that reasonably apply.2383 - Do not include markdown formatting or code fences in your response.2384 """;23852386 // Send the request to Claude for content moderation2387 var response = await client.Messages.Create(2388 new()2389 {2390 Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs2391 MaxTokens = 2048, // Increased max token count to handle batches2392 Messages = [new() { Role = Role.User, Content = assessmentPrompt }],2393 }2394 );23952396 // Narrow the first content block to a text block, then parse Claude's JSON response2397 if (!response.Content[0].TryPickText(out var textBlock))2398 {2399 throw new InvalidOperationException("Expected a text response from Claude.");2400 }2401 return JsonNode.Parse(textBlock.Text)!;2402 }24032404 // Process the batch of comments and get the response2405 var moderationResults = await BatchModerateMessages(userComments, unsafeCategories);24062407 // Print the results for each detected violation2408 foreach (var violation in moderationResults["violations"]!.AsArray())2409 {2410 var flaggedComment = userComments[violation!["id"]!.GetValue<int>()];2411 var violatedCategories = string.Join(2412 ", ",2413 violation["categories"]!.AsArray().Select(category => category!.GetValue<string>())2414 );2415 var explanation = violation["explanation"]!.GetValue<string>();24162417 Console.WriteLine($"""2418 Comment: {flaggedComment}2419 Violated Categories: {violatedCategories}2420 Explanation: {explanation}24212422 """);2423 }2424 ```24252426 ```go Go2427 // batchViolation is one entry in Claude's "violations" array: the index of the2428 // offending message plus the categories it violated and why.2429 type batchViolation struct {2430 ID int `json:"id"`2431 Categories []string `json:"categories"`2432 Explanation string `json:"explanation"`2433 }24342435 func batchModerateMessages(messages []string, unsafeCategories []string) []batchViolation {2436 // Convert the unsafe categories into a string, with each category on a new line2437 unsafeCategoryStr := strings.Join(unsafeCategories, "\n")24382439 // Format messages string, with each message wrapped in XML-like tags and given an ID2440 messageLines := make([]string, len(messages))2441 for i, message := range messages {2442 messageLines[i] = fmt.Sprintf("<message id=%d>%s</message>", i, message)2443 }2444 messagesStr := strings.Join(messageLines, "\n")24452446 // Construct the prompt for Claude, including the messages and unsafe categories2447 assessmentPrompt := fmt.Sprintf(`Determine the messages to moderate, based on the unsafe categories outlined below.24482449 Messages:2450 <messages>2451 %s2452 </messages>24532454 Unsafe Categories:2455 <categories>2456 %s2457 </categories>24582459 Respond with ONLY a JSON object, using the format below:2460 {2461 "violations": [2462 {2463 "id": <message id>,2464 "categories": [list of violated categories],2465 "explanation": <Explanation of why there's a violation>2466 }2467 ]2468 }24692470 Important Notes:2471 - Remember to analyze every message for a violation.2472 - Select any number of violations that reasonably apply.2473 - Do not include markdown formatting or code fences in your response.`, messagesStr, unsafeCategoryStr)24742475 // Send the request to Claude for content moderation2476 response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{2477 Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs2478 MaxTokens: 2048, // Increased max token count to handle batches2479 Messages: []anthropic.MessageParam{2480 anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),2481 },2482 })2483 if err != nil {2484 log.Fatal(err)2485 }24862487 // Narrow the first content block to a text block before reading its text2488 textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)2489 if !ok {2490 log.Fatalf("expected a text block, got %q", response.Content[0].Type)2491 }24922493 // Parse the JSON response from Claude2494 var assessment struct {2495 Violations []batchViolation `json:"violations"`2496 }2497 if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {2498 log.Fatal(err)2499 }2500 return assessment.Violations2501 }25022503 // moderateAllCommentsAsBatch moderates the whole batch of comments in a single2504 // request and prints the results for each detected violation.2505 func moderateAllCommentsAsBatch() {2506 // Process the batch of comments and get the response2507 violations := batchModerateMessages(userComments, unsafeCategories)25082509 // Print the results for each detected violation2510 for _, violation := range violations {2511 fmt.Printf(`Comment: %s2512 Violated Categories: %s2513 Explanation: %s25142515 `, userComments[violation.ID], strings.Join(violation.Categories, ", "), violation.Explanation)2516 }2517 }25182519 ```25202521 ```java Java2522 JsonNode batchModerateMessages(List<String> messages, List<String> unsafeCategories)2523 throws JsonProcessingException {2524 // Convert the unsafe categories into a string, with each category on a new line2525 String unsafeCategoryStr = String.join("\n", unsafeCategories);25262527 // Format messages string, with each message wrapped in XML-like tags and given an ID2528 String messagesStr = IntStream.range(0, messages.size())2529 .mapToObj(idx -> "<message id=%d>%s</message>".formatted(idx, messages.get(idx)))2530 .collect(Collectors.joining("\n"));25312532 // Construct the prompt for Claude, including the messages and unsafe categories2533 String assessmentPrompt = """2534 Determine the messages to moderate, based on the unsafe categories outlined below.25352536 Messages:2537 <messages>2538 %s2539 </messages>25402541 Unsafe Categories:2542 <categories>2543 %s2544 </categories>25452546 Respond with ONLY a JSON object, using the format below:2547 {2548 "violations": [2549 {2550 "id": <message id>,2551 "categories": [list of violated categories],2552 "explanation": <Explanation of why there's a violation>2553 }2554 ]2555 }25562557 Important Notes:2558 - Remember to analyze every message for a violation.2559 - Select any number of violations that reasonably apply.2560 - Do not include markdown formatting or code fences in your response."""2561 .formatted(messagesStr, unsafeCategoryStr);25622563 // Send the request to Claude for content moderation2564 Message response = client.messages().create(MessageCreateParams.builder()2565 .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs2566 .maxTokens(2048) // Increased max token count to handle batches2567 .addUserMessage(assessmentPrompt)2568 .build());25692570 // Parse the JSON response from Claude2571 String assessmentJson = response.content().stream()2572 .flatMap(contentBlock -> contentBlock.text().stream())2573 .findFirst()2574 .orElseThrow()2575 .text();2576 return new ObjectMapper().readTree(assessmentJson);2577 }25782579 // Process the batch of comments and print the results for each detected violation2580 void printBatchViolations() throws JsonProcessingException {2581 JsonNode response = batchModerateMessages(userComments, unsafeCategories);25822583 ObjectMapper mapper = new ObjectMapper();2584 for (JsonNode violation : response.required("violations")) {2585 List<String> violatedCategories =2586 mapper.convertValue(violation.required("categories"), new TypeReference<List<String>>() {});2587 IO.println("""2588 Comment: %s2589 Violated Categories: %s2590 Explanation: %s2591 """.formatted(2592 userComments.get(violation.required("id").asInt()),2593 String.join(", ", violatedCategories),2594 violation.required("explanation").asText()));2595 }2596 }2597 ```25982599 ```php PHP2600 $batchModerateMessages = function (array $messages, array $unsafeCategories) use ($client): array {2601 // Convert the unsafe categories into a string, with each category on a new line2602 $unsafeCategoryStr = implode("\n", $unsafeCategories);26032604 // Format the messages string, with each message wrapped in XML-like tags and given an ID2605 $messageLines = [];2606 foreach ($messages as $idx => $msg) {2607 $messageLines[] = "<message id={$idx}>{$msg}</message>";2608 }2609 $messagesStr = implode("\n", $messageLines);26102611 // Construct the prompt for Claude, including the messages and unsafe categories2612 $assessmentPrompt = <<<PROMPT2613 Determine the messages to moderate, based on the unsafe categories outlined below.26142615 Messages:2616 <messages>2617 {$messagesStr}2618 </messages>26192620 Unsafe Categories:2621 <categories>2622 {$unsafeCategoryStr}2623 </categories>26242625 Respond with ONLY a JSON object, using the format below:2626 {2627 "violations": [2628 {2629 "id": <message id>,2630 "categories": [list of violated categories],2631 "explanation": <Explanation of why there's a violation>2632 }2633 ]2634 }26352636 Important Notes:2637 - Remember to analyze every message for a violation.2638 - Select any number of violations that reasonably apply.2639 - Do not include markdown formatting or code fences in your response.2640 PROMPT;26412642 // Send the request to Claude for content moderation2643 $response = $client->messages->create(2644 model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs2645 maxTokens: 2048, // Increased max token count to handle batches2646 messages: [['role' => 'user', 'content' => $assessmentPrompt]],2647 );26482649 // Parse the JSON response from Claude. The SDK decodes each content block2650 // into its concrete class, so find the TextBlock before reading the text.2651 $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)2652 ?? throw new RuntimeException('Expected a text block in the response.');26532654 return json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);2655 };26562657 // Process the batch of comments and get the response2658 $responseObj = $batchModerateMessages($userComments, $unsafeCategories);26592660 // Print the results for each detected violation2661 foreach ($responseObj['violations'] as $violation) {2662 echo "Comment: {$userComments[$violation['id']]}\n";2663 echo 'Violated Categories: ' . implode(', ', $violation['categories']) . "\n";2664 echo "Explanation: {$violation['explanation']}\n\n";2665 }2666 ```26672668 ```ruby Ruby2669 def batch_moderate_messages(messages, unsafe_categories)2670 # Convert the unsafe categories into a string, with each category on a new line2671 unsafe_category_str = unsafe_categories.join("\n")26722673 # Format the messages string, with each message wrapped in XML-like tags and given an ID2674 messages_str = messages2675 .map.with_index { |message, index| "<message id=#{index}>#{message}</message>" }2676 .join("\n")26772678 # Construct the prompt for Claude, including the messages and unsafe categories2679 assessment_prompt = <<~PROMPT.chomp2680 Determine the messages to moderate, based on the unsafe categories outlined below.26812682 Messages:2683 <messages>2684 #{messages_str}2685 </messages>26862687 Unsafe Categories:2688 <categories>2689 #{unsafe_category_str}2690 </categories>26912692 Respond with ONLY a JSON object, using the format below:2693 {2694 "violations": [2695 {2696 "id": <message id>,2697 "categories": [list of violated categories],2698 "explanation": <Explanation of why there's a violation>2699 }2700 ]2701 }27022703 Important Notes:2704 - Remember to analyze every message for a violation.2705 - Select any number of violations that reasonably apply.2706 - Do not include markdown formatting or code fences in your response.2707 PROMPT27082709 # Send the request to Claude for content moderation2710 response = CLIENT.messages.create(2711 model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs2712 max_tokens: 2048, # Increased max token count to handle batches2713 messages: [{role: :user, content: assessment_prompt}]2714 )27152716 # Parse the JSON response from Claude2717 text_block = response.content.find { it.type == :text }2718 JSON.parse(text_block.text)2719 end272027212722 # Process the batch of comments and get the response2723 response_obj = batch_moderate_messages(USER_COMMENTS, UNSAFE_CATEGORIES)27242725 # Print the results for each detected violation2726 response_obj["violations"].each do |violation|2727 puts <<~RESULT2728 Comment: #{USER_COMMENTS[violation["id"]]}2729 Violated Categories: #{violation["categories"].join(", ")}2730 Explanation: #{violation["explanation"]}27312732 RESULT2733 end2734 ```2735</CodeGroup>27362737In this example, the `batch_moderate_messages` function handles the moderation of an entire batch of messages with a single Claude API call. Inside the function, a prompt is created that includes the list of messages to evaluate and the unsafe content categories. The prompt directs Claude to return a JSON object listing all messages that contain violations. Each message in the response is identified by its `id`, which corresponds to the message's position in the batch. Keep in mind that finding the optimal batch size for your specific needs may require some experimentation. While larger batch sizes can lower costs, they might also lead to a slight decrease in quality. Additionally, you may need to increase the `max_tokens` parameter in the Claude API call to accommodate longer responses. For details on the maximum number of tokens your chosen model can output, refer to the [model comparison table](https://platform.claude.com/docs/en/models/overview#latest-models-comparison).27382739<CardGroup cols={2}>2740 <Card title="Content moderation cookbook" icon="link" href="https://platform.claude.com/cookbook/misc-building-moderation-filter">2741 View a fully implemented code-based example of how to use Claude for content moderation.2742 </Card>27432744 <Card title="Mitigate jailbreaks" icon="link" href="https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks">2745 Explore guardrail techniques to moderate interactions with Claude.2746 </Card>2747</CardGroup>2748