SPB Git

Repository Public

content-moderation.md 112.5 KB 2,748 lines markdown

1---2title: Content moderation3url: https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation4description: Content moderation is a critical aspect of maintaining a safe, respectful, and productive environment in digital applications. This guide discusses how Claude can be used to moderate content within your digital application.5---67> Visit the [content moderation cookbook](https://platform.claude.com/cookbook/misc-building-moderation-filter) to see an example content moderation implementation using Claude.89<Tip>10  This guide is focused on moderating user-generated content within your application. If you're looking for guidance on moderating interactions with Claude, refer to 1112  [Mitigate jailbreaks and prompt injections](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks)1314  .15</Tip>1617## Before building with Claude1819### Decide whether to use Claude for content moderation2021Here are some key indicators that you should use an LLM like Claude instead of a traditional ML or rules-based approach for content moderation:2223<AccordionGroup>24  <Accordion title="You want a cost-effective and rapid implementation">25    Traditional ML methods require significant engineering resources, ML expertise, and infrastructure costs. Human moderation systems incur even higher costs. With Claude, you can have a sophisticated moderation system operational in significantly less time and at a much lower cost.26  </Accordion>2728  <Accordion title="You want both semantic understanding and quick decisions">29    Traditional ML approaches, such as bag-of-words models or simple pattern matching, often struggle to understand the tone, intent, and context of the content. While human moderation systems excel at understanding semantic meaning, they require time for content to be reviewed. Claude addresses both needs by combining semantic understanding with the ability to deliver moderation decisions quickly.30  </Accordion>3132  <Accordion title="You need consistent policy decisions">33    By leveraging its advanced reasoning capabilities, Claude can interpret and apply complex moderation guidelines uniformly. This consistency helps ensure fair treatment of all content, reducing the risk of inconsistent or biased moderation decisions that can undermine user trust.34  </Accordion>3536  <Accordion title="Your moderation policies are likely to change or evolve over time">37    Once a traditional ML approach has been established, changing it is a laborious and data-intensive undertaking. On the other hand, as your product or customer needs evolve, Claude can easily adapt to changes or additions to moderation policies without extensive relabeling of training data.38  </Accordion>3940  <Accordion title="You require interpretable reasoning for your moderation decisions">41    If you want to provide users or regulators with clear explanations behind moderation decisions, Claude can generate detailed and coherent justifications. This transparency is important for building trust and ensuring accountability in content moderation practices.42  </Accordion>4344  <Accordion title="You need multilingual support without maintaining separate models">45    Traditional ML approaches typically require separate models or extensive translation processes for each supported language. Human moderation requires hiring a workforce fluent in each supported language. Claude’s multilingual capabilities allow it to classify tickets in various languages without the need for separate models or extensive translation processes, streamlining moderation for global customer bases.46  </Accordion>4748  <Accordion title="You require multimodal support">49    Claude's multimodal capabilities allow it to analyze and interpret content across both text and images. This makes it a versatile tool for comprehensive content moderation in environments where different media types need to be evaluated together.50  </Accordion>51</AccordionGroup>5253<Note>54  All Claude models are trained with built-in safety behaviors. This may result in Claude moderating content deemed particularly dangerous (in line with the 5556  [Acceptable Use Policy](https://www.anthropic.com/legal/aup)5758  ), regardless of the prompt used. For example, an adult website that wants to allow users to post explicit sexual content may find that Claude still flags explicit content as requiring moderation, even if they specify in their prompt not to moderate explicit sexual content. Consider reviewing the AUP in advance of building a moderation solution.59</Note>6061### Generate examples of content to moderate6263Before developing a content moderation solution, first create examples of content that should be flagged and content that should not be flagged. Ensure that you include edge cases and challenging scenarios that may be difficult for a content moderation system to handle effectively. Afterward, review your examples to create a well-defined list of moderation categories. For instance, the examples generated by a social media platform might include the following:6465<CodeGroup exclude="shell">66  ```python Python67  client = anthropic.Anthropic()6869  allowed_user_comments = [70      "This movie was great, I really enjoyed it. The main actor really killed it!",71      "I hate Mondays.",72      "It is a great time to invest in gold!",73  ]7475  disallowed_user_comments = [76      "Delete this post now or you better hide. I am coming after you and your family.",77      "Stay away from the 5G cellphones!! They are using 5G to control you.",78      "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",79  ]8081  # Sample user comments to test the content moderation82  user_comments = allowed_user_comments + disallowed_user_comments8384  # Categories considered unsafe for content moderation85  unsafe_categories = [86      "Child Exploitation",87      "Conspiracy Theories",88      "Hate",89      "Indiscriminate Weapons",90      "Intellectual Property",91      "Non-Violent Crimes",92      "Privacy",93      "Self-Harm",94      "Sex Crimes",95      "Sexual Content",96      "Specialized Advice",97      "Violent Crimes",98  ]99  ```100101  ```typescript TypeScript102  const client = new Anthropic();103104  const allowedUserComments = [105    "This movie was great, I really enjoyed it. The main actor really killed it!",106    "I hate Mondays.",107    "It is a great time to invest in gold!"108  ];109110  const disallowedUserComments = [111    "Delete this post now or you better hide. I am coming after you and your family.",112    "Stay away from the 5G cellphones!! They are using 5G to control you.",113    "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"114  ];115116  // Sample user comments to test the content moderation117  const userComments = [...allowedUserComments, ...disallowedUserComments];118119  // Categories considered unsafe for content moderation120  const unsafeCategories = [121    "Child Exploitation",122    "Conspiracy Theories",123    "Hate",124    "Indiscriminate Weapons",125    "Intellectual Property",126    "Non-Violent Crimes",127    "Privacy",128    "Self-Harm",129    "Sex Crimes",130    "Sexual Content",131    "Specialized Advice",132    "Violent Crimes"133  ];134  ```135136  ```csharp C#137  var client = new AnthropicClient();138139  string[] allowedUserComments =140  [141      "This movie was great, I really enjoyed it. The main actor really killed it!",142      "I hate Mondays.",143      "It is a great time to invest in gold!",144  ];145146  string[] disallowedUserComments =147  [148      "Delete this post now or you better hide. I am coming after you and your family.",149      "Stay away from the 5G cellphones!! They are using 5G to control you.",150      "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",151  ];152153  // Sample user comments to test the content moderation154  string[] userComments = [.. allowedUserComments, .. disallowedUserComments];155156  // Categories considered unsafe for content moderation157  string[] unsafeCategories =158  [159      "Child Exploitation",160      "Conspiracy Theories",161      "Hate",162      "Indiscriminate Weapons",163      "Intellectual Property",164      "Non-Violent Crimes",165      "Privacy",166      "Self-Harm",167      "Sex Crimes",168      "Sexual Content",169      "Specialized Advice",170      "Violent Crimes",171  ];172  ```173174  ```go Go175  var client = anthropic.NewClient()176177  var allowedUserComments = []string{178  	"This movie was great, I really enjoyed it. The main actor really killed it!",179  	"I hate Mondays.",180  	"It is a great time to invest in gold!",181  }182183  var disallowedUserComments = []string{184  	"Delete this post now or you better hide. I am coming after you and your family.",185  	"Stay away from the 5G cellphones!! They are using 5G to control you.",186  	"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",187  }188189  // Sample user comments to test the content moderation190  var userComments = slices.Concat(allowedUserComments, disallowedUserComments)191192  // Categories considered unsafe for content moderation193  var unsafeCategories = []string{194  	"Child Exploitation",195  	"Conspiracy Theories",196  	"Hate",197  	"Indiscriminate Weapons",198  	"Intellectual Property",199  	"Non-Violent Crimes",200  	"Privacy",201  	"Self-Harm",202  	"Sex Crimes",203  	"Sexual Content",204  	"Specialized Advice",205  	"Violent Crimes",206  }207208  ```209210  ```java Java211  final AnthropicClient client = AnthropicOkHttpClient.fromEnv();212213  final List<String> allowedUserComments = List.of(214          "This movie was great, I really enjoyed it. The main actor really killed it!",215          "I hate Mondays.",216          "It is a great time to invest in gold!");217218  final List<String> disallowedUserComments = List.of(219          "Delete this post now or you better hide. I am coming after you and your family.",220          "Stay away from the 5G cellphones!! They are using 5G to control you.",221          "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!");222223  // Sample user comments to test the content moderation224  final List<String> userComments =225          Stream.concat(allowedUserComments.stream(), disallowedUserComments.stream()).toList();226227  // Categories considered unsafe for content moderation228  final List<String> unsafeCategories = List.of(229          "Child Exploitation",230          "Conspiracy Theories",231          "Hate",232          "Indiscriminate Weapons",233          "Intellectual Property",234          "Non-Violent Crimes",235          "Privacy",236          "Self-Harm",237          "Sex Crimes",238          "Sexual Content",239          "Specialized Advice",240          "Violent Crimes");241  ```242243  ```php PHP244  $client = new Client();245246  $allowedUserComments = [247      'This movie was great, I really enjoyed it. The main actor really killed it!',248      'I hate Mondays.',249      'It is a great time to invest in gold!',250  ];251252  $disallowedUserComments = [253      'Delete this post now or you better hide. I am coming after you and your family.',254      'Stay away from the 5G cellphones!! They are using 5G to control you.',255      'Congratulations! You have won a $1,000 gift card. Click here to claim your prize!',256  ];257258  // Sample user comments to test the content moderation259  $userComments = [...$allowedUserComments, ...$disallowedUserComments];260261  // Categories considered unsafe for content moderation262  $unsafeCategories = [263      'Child Exploitation',264      'Conspiracy Theories',265      'Hate',266      'Indiscriminate Weapons',267      'Intellectual Property',268      'Non-Violent Crimes',269      'Privacy',270      'Self-Harm',271      'Sex Crimes',272      'Sexual Content',273      'Specialized Advice',274      'Violent Crimes',275  ];276  ```277278  ```ruby Ruby279  CLIENT = Anthropic::Client.new280281  ALLOWED_USER_COMMENTS = [282    "This movie was great, I really enjoyed it. The main actor really killed it!",283    "I hate Mondays.",284    "It is a great time to invest in gold!"285  ]286287  DISALLOWED_USER_COMMENTS = [288    "Delete this post now or you better hide. I am coming after you and your family.",289    "Stay away from the 5G cellphones!! They are using 5G to control you.",290    "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"291  ]292293  # Sample user comments to test the content moderation294  USER_COMMENTS = ALLOWED_USER_COMMENTS + DISALLOWED_USER_COMMENTS295296  # Categories considered unsafe for content moderation297  UNSAFE_CATEGORIES = [298    "Child Exploitation",299    "Conspiracy Theories",300    "Hate",301    "Indiscriminate Weapons",302    "Intellectual Property",303    "Non-Violent Crimes",304    "Privacy",305    "Self-Harm",306    "Sex Crimes",307    "Sexual Content",308    "Specialized Advice",309    "Violent Crimes"310  ]311  ```312</CodeGroup>313314Effectively moderating these examples requires a nuanced understanding of language. In the comment, `This movie was great, I really enjoyed it. The main actor really killed it!`, the content moderation system needs to recognize that "killed it" is a metaphor, not an indication of actual violence. Conversely, despite the lack of explicit mentions of violence, the comment `Delete this post now or you better hide. I am coming after you and your family.` should be flagged by the content moderation system.315316The unsafe categories can be customized to fit your specific needs. For example, if you want to prevent minors from creating content on your website, you could add "Underage Posting" to the categories.317318***319320## How to moderate content using Claude321322### Select the right Claude model323324When selecting a model, it’s important to consider the size of your data. If costs are a concern, a smaller model such as Claude Haiku 4.5 is an excellent choice because of its cost-effectiveness. The following is an estimate of the cost to moderate text for a social media platform that receives one billion posts per month:325326* **Content size**327328  * Posts per month: 1B329  * Characters per post: 100330  * Total characters: 100B331332* **Estimated tokens**333334  * Input tokens: 28.6B (assuming 1 token per 3.5 characters)335  * Percentage of messages flagged: 3%336  * Output tokens per flagged message: 50337  * Total output tokens: 1.5B338339* **Claude Haiku 4.5 estimated cost**340341  * Input token cost: 28,600 MTok \* $1.00/MTok = $28,600 USD342  * Output token cost: 1,500 MTok \* $5.00/MTok = $7,500 USD343  * Monthly cost: $28,600 + $7,500 = $36,100 USD344345* **Claude Opus 5 estimated cost**346347  * Input token cost: 28,600 MTok \* $5.00/MTok = $143,000 USD348  * Output token cost: 1,500 MTok \* $25.00/MTok = $37,500 USD349  * Monthly cost: $143,000 + $37,500 = $180,500 USD350351* **Claude Opus 4.8 estimated cost**352353  * Input token cost: 28,600 MTok \* $5.00/MTok = $143,000 USD354  * Output token cost: 1,500 MTok \* $25.00/MTok = $37,500 USD355  * Monthly cost: $143,000 + $37,500 = $180,500 USD356357<Tip>358  Actual costs may differ from these estimates. These estimates are based on the prompt highlighted in the section on 359360  [batch processing](https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation#consider-batch-processing)361362  . Output tokens can be reduced even further by removing the 363364  `explanation`365366   field from the response.367</Tip>368369### Build a strong prompt370371To use Claude for content moderation, Claude must understand the moderation requirements of your application. Start by writing a prompt that allows you to define your moderation needs:372373<CodeGroup exclude="shell">374  ```python Python375  def moderate_message(message, unsafe_categories):376      # Convert the unsafe categories into a string, with each category on a new line377      unsafe_category_str = "\n".join(unsafe_categories)378379      # Construct the prompt for Claude, including the message and unsafe categories380      assessment_prompt = f"""381      Determine whether the following message warrants moderation,382      based on the unsafe categories outlined below.383384      Message:385      <message>{message}</message>386387      Unsafe Categories:388      <categories>389      {unsafe_category_str}390      </categories>391392      Respond with ONLY a JSON object, using the format below:393      {{394      "violation": <Boolean field denoting whether the message should be moderated>,395      "categories": [Comma-separated list of violated categories],396      "explanation": [Optional. Only include if there is a violation.]397      }}398  Do not include markdown formatting or code fences in your response."""399400      # Send the request to Claude for content moderation401      response = client.messages.create(402          model="claude-haiku-4-5-20251001",  # Using the Haiku model for lower costs403          max_tokens=200,404          messages=[{"role": "user", "content": assessment_prompt}],405      )406407      # Parse the JSON response from Claude408      text_block = next(block for block in response.content if block.type == "text")409      assessment = json.loads(text_block.text)410411      # Extract the violation status from the assessment412      contains_violation = assessment["violation"]413414      # If there's a violation, get the categories and explanation; otherwise, use empty defaults415      violated_categories = assessment.get("categories", []) if contains_violation else []416      explanation = assessment.get("explanation") if contains_violation else None417418      return contains_violation, violated_categories, explanation419420421  # Process each comment and print the results422  for comment in user_comments:423      print(f"\nComment: {comment}")424      violation, violated_categories, explanation = moderate_message(425          comment, unsafe_categories426      )427428      if violation:429          print(f"Violated Categories: {', '.join(violated_categories)}")430          print(f"Explanation: {explanation}")431      else:432          print("No issues detected.")433  ```434435  ```typescript TypeScript436  // Shape of the JSON assessment Claude returns437  interface ModerationAssessment {438    violation: boolean;439    categories?: string[];440    explanation?: string;441  }442443  async function moderateMessage(444    message: string,445    unsafeCategories: string[]446  ): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {447    // Convert the unsafe categories into a string, with each category on a new line448    const unsafeCategoryStr = unsafeCategories.join("\n");449450    // Construct the prompt for Claude, including the message and unsafe categories451    const assessmentPrompt = `452      Determine whether the following message warrants moderation,453      based on the unsafe categories outlined below.454455      Message:456      <message>${message}</message>457458      Unsafe Categories:459      <categories>460      ${unsafeCategoryStr}461      </categories>462463      Respond with ONLY a JSON object, using the format below:464      {465      "violation": <Boolean field denoting whether the message should be moderated>,466      "categories": [Comma-separated list of violated categories],467      "explanation": [Optional. Only include if there is a violation.]468      }469  Do not include markdown formatting or code fences in your response.`;470471    // Send the request to Claude for content moderation472    const response = await client.messages.create({473      model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs474      max_tokens: 200,475      messages: [{ role: "user", content: assessmentPrompt }]476    });477478    // Parse the JSON response from Claude479    const textBlock = response.content.find((block) => block.type === "text");480    if (!textBlock) {481      throw new Error("Expected a text block in the response");482    }483    const assessment: ModerationAssessment = JSON.parse(textBlock.text);484485    // Extract the violation status from the assessment486    const containsViolation = assessment.violation;487488    // If there's a violation, get the categories and explanation; otherwise, use empty defaults489    const violatedCategories = containsViolation ? assessment.categories ?? [] : [];490    const explanation = containsViolation ? assessment.explanation : undefined;491492    return { violation: containsViolation, violatedCategories, explanation };493  }494495  // Process each comment and print the results496  for (const comment of userComments) {497    console.log(`\nComment: ${comment}`);498    const { violation, violatedCategories, explanation } = await moderateMessage(499      comment,500      unsafeCategories501    );502503    if (violation) {504      console.log(`Violated Categories: ${violatedCategories.join(", ")}`);505      console.log(`Explanation: ${explanation}`);506    } else {507      console.log("No issues detected.");508    }509  }510  ```511512  ```csharp C#513  async Task<(bool ContainsViolation, List<string> ViolatedCategories, string? Explanation)> ModerateMessage(514      string message,515      IReadOnlyList<string> categories516  )517  {518      // Convert the unsafe categories into a string, with each category on a new line519      var unsafeCategoryText = string.Join("\n", categories);520521      // Construct the prompt for Claude, including the message and unsafe categories522      var assessmentPrompt = $$"""523524      Determine whether the following message warrants moderation,525      based on the unsafe categories outlined below.526527      Message:528      <message>{{message}}</message>529530      Unsafe Categories:531      <categories>532      {{unsafeCategoryText}}533      </categories>534535      Respond with ONLY a JSON object, using the format below:536      {537      "violation": <Boolean field denoting whether the message should be moderated>,538      "categories": [Comma-separated list of violated categories],539      "explanation": [Optional. Only include if there is a violation.]540      }541  Do not include markdown formatting or code fences in your response.542  """;543544      // Send the request to Claude for content moderation545      var response = await client.Messages.Create(546          new()547          {548              Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs549              MaxTokens = 200,550              Messages = [new() { Role = Role.User, Content = assessmentPrompt }],551          }552      );553554      // Narrow the first content block to a text block, then parse Claude's JSON response555      if (!response.Content[0].TryPickText(out var textBlock))556      {557          throw new InvalidOperationException("Expected a text response from Claude.");558      }559      var assessment = JsonNode.Parse(textBlock.Text)!;560561      // Extract the violation status from the assessment562      var containsViolation = assessment["violation"]!.GetValue<bool>();563564      // If there's a violation, get the categories and explanation; otherwise, use empty defaults565      List<string> violatedCategories = containsViolation566          ? assessment["categories"]?.AsArray().Select(category => category!.GetValue<string>()).ToList() ?? []567          : [];568      var explanation = containsViolation ? assessment["explanation"]?.GetValue<string>() : null;569570      return (containsViolation, violatedCategories, explanation);571  }572573  // Process each comment and print the results574  foreach (var comment in userComments)575  {576      Console.WriteLine($"\nComment: {comment}");577      var (violation, violatedCategories, explanation) = await ModerateMessage(comment, unsafeCategories);578579      if (violation)580      {581          Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");582          Console.WriteLine($"Explanation: {explanation}");583      }584      else585      {586          Console.WriteLine("No issues detected.");587      }588  }589  ```590591  ```go Go592  func moderateMessage(message string, unsafeCategories []string) (bool, []string, string) {593  	// Convert the unsafe categories into a string, with each category on a new line594  	unsafeCategoryStr := strings.Join(unsafeCategories, "\n")595596  	// Construct the prompt for Claude, including the message and unsafe categories597  	assessmentPrompt := fmt.Sprintf(`598      Determine whether the following message warrants moderation,599      based on the unsafe categories outlined below.600601      Message:602      <message>%s</message>603604      Unsafe Categories:605      <categories>606      %s607      </categories>608609      Respond with ONLY a JSON object, using the format below:610      {611      "violation": <Boolean field denoting whether the message should be moderated>,612      "categories": [Comma-separated list of violated categories],613      "explanation": [Optional. Only include if there is a violation.]614      }615  Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)616617  	// Send the request to Claude for content moderation618  	response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{619  		Model:     anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs620  		MaxTokens: 200,621  		Messages: []anthropic.MessageParam{622  			anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),623  		},624  	})625  	if err != nil {626  		log.Fatal(err)627  	}628629  	// Narrow the first content block to a text block before reading its text630  	textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)631  	if !ok {632  		log.Fatalf("expected a text block, got %q", response.Content[0].Type)633  	}634635  	// Parse the JSON response from Claude636  	var assessment struct {637  		Violation   bool     `json:"violation"`638  		Categories  []string `json:"categories"`639  		Explanation string   `json:"explanation"`640  	}641  	if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {642  		log.Fatal(err)643  	}644645  	// If there's a violation, return the categories and explanation; otherwise, use empty defaults646  	if !assessment.Violation {647  		return false, nil, ""648  	}649  	return true, assessment.Categories, assessment.Explanation650  }651652  // moderateAllComments processes each comment and prints the results.653  func moderateAllComments() {654  	for _, comment := range userComments {655  		fmt.Printf("\nComment: %s\n", comment)656  		violation, violatedCategories, explanation := moderateMessage(comment, unsafeCategories)657658  		if violation {659  			fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))660  			fmt.Printf("Explanation: %s\n", explanation)661  		} else {662  			fmt.Println("No issues detected.")663  		}664  	}665  }666667  ```668669  ```java Java670  record ModerationResult(boolean violation, List<String> violatedCategories, String explanation) {}671672  ModerationResult moderateMessage(String message, List<String> unsafeCategories)673          throws JsonProcessingException {674      // Convert the unsafe categories into a string, with each category on a new line675      String unsafeCategoryStr = String.join("\n", unsafeCategories);676677      // Construct the prompt for Claude, including the message and unsafe categories678      String assessmentPrompt = """679680              Determine whether the following message warrants moderation,681              based on the unsafe categories outlined below.682683              Message:684              <message>%s</message>685686              Unsafe Categories:687              <categories>688              %s689              </categories>690691              Respond with ONLY a JSON object, using the format below:692              {693              "violation": <Boolean field denoting whether the message should be moderated>,694              "categories": [Comma-separated list of violated categories],695              "explanation": [Optional. Only include if there is a violation.]696              }697          Do not include markdown formatting or code fences in your response."""698              .formatted(message, unsafeCategoryStr);699700      // Send the request to Claude for content moderation701      Message response = client.messages().create(MessageCreateParams.builder()702              .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs703              .maxTokens(200)704              .addUserMessage(assessmentPrompt)705              .build());706707      // Parse the JSON response from Claude708      String assessmentJson = response.content().stream()709              .flatMap(contentBlock -> contentBlock.text().stream())710              .findFirst()711              .orElseThrow()712              .text();713      ObjectMapper mapper = new ObjectMapper();714      JsonNode assessment = mapper.readTree(assessmentJson);715716      // Extract the violation status from the assessment717      boolean containsViolation = assessment.required("violation").asBoolean();718719      // If there's a violation, get the categories and explanation; otherwise, use empty defaults720      List<String> violatedCategories = containsViolation && assessment.has("categories")721              ? mapper.convertValue(assessment.get("categories"), new TypeReference<List<String>>() {})722              : List.of();723      String explanation = containsViolation && assessment.hasNonNull("explanation")724              ? assessment.get("explanation").asText()725              : null;726727      return new ModerationResult(containsViolation, violatedCategories, explanation);728  }729730  // Process each comment and print the results731  void printModerationResults() throws JsonProcessingException {732      for (String comment : userComments) {733          IO.println("\nComment: " + comment);734          ModerationResult result = moderateMessage(comment, unsafeCategories);735736          if (result.violation()) {737              IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));738              IO.println("Explanation: " + result.explanation());739          } else {740              IO.println("No issues detected.");741          }742      }743  }744  ```745746  ```php PHP747  $moderateMessage = function (string $message, array $unsafeCategories) use ($client): array {748      // Convert the unsafe categories into a string, with each category on a new line749      $unsafeCategoryStr = implode("\n", $unsafeCategories);750751      // Construct the prompt for Claude, including the message and unsafe categories752      $assessmentPrompt = <<<PROMPT753754          Determine whether the following message warrants moderation,755          based on the unsafe categories outlined below.756757          Message:758          <message>{$message}</message>759760          Unsafe Categories:761          <categories>762          {$unsafeCategoryStr}763          </categories>764765          Respond with ONLY a JSON object, using the format below:766          {767          "violation": <Boolean field denoting whether the message should be moderated>,768          "categories": [Comma-separated list of violated categories],769          "explanation": [Optional. Only include if there is a violation.]770          }771      Do not include markdown formatting or code fences in your response.772      PROMPT;773774      // Send the request to Claude for content moderation775      $response = $client->messages->create(776          model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs777          maxTokens: 200,778          messages: [['role' => 'user', 'content' => $assessmentPrompt]],779      );780781      // Parse the JSON response from Claude. The SDK decodes each content block782      // into its concrete class, so find the TextBlock before reading the text.783      $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)784          ?? throw new RuntimeException('Expected a text block in the response.');785      $assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);786787      // Extract the violation status from the assessment788      $containsViolation = $assessment['violation'];789790      // If there's a violation, get the categories and explanation; otherwise, use empty defaults791      $violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];792      $explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;793794      return [$containsViolation, $violatedCategories, $explanation];795  };796797  // Process each comment and print the results798  foreach ($userComments as $comment) {799      echo "\nComment: {$comment}\n";800      [$violation, $violatedCategories, $explanation] = $moderateMessage($comment, $unsafeCategories);801802      if ($violation) {803          echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";804          echo "Explanation: {$explanation}\n";805      } else {806          echo "No issues detected.\n";807      }808  }809  ```810811  ```ruby Ruby812  def moderate_message(message, unsafe_categories)813    # Convert the unsafe categories into a string, with each category on a new line814    unsafe_category_str = unsafe_categories.join("\n")815816    # Construct the prompt for Claude, including the message and unsafe categories817    assessment_prompt = <<~PROMPT.chomp818819          Determine whether the following message warrants moderation,820          based on the unsafe categories outlined below.821822          Message:823          <message>#{message}</message>824825          Unsafe Categories:826          <categories>827          #{unsafe_category_str}828          </categories>829830          Respond with ONLY a JSON object, using the format below:831          {832          "violation": <Boolean field denoting whether the message should be moderated>,833          "categories": [Comma-separated list of violated categories],834          "explanation": [Optional. Only include if there is a violation.]835          }836      Do not include markdown formatting or code fences in your response.837    PROMPT838839    # Send the request to Claude for content moderation840    response = CLIENT.messages.create(841      model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs842      max_tokens: 200,843      messages: [{role: :user, content: assessment_prompt}]844    )845846    # Parse the JSON response from Claude847    text_block = response.content.find { it.type == :text }848    assessment = JSON.parse(text_block.text)849850    # Extract the violation status from the assessment851    contains_violation = assessment["violation"]852853    # If there's a violation, get the categories and explanation; otherwise, use empty defaults854    violated_categories = contains_violation ? assessment.fetch("categories", []) : []855    explanation = contains_violation ? assessment["explanation"] : nil856857    [contains_violation, violated_categories, explanation]858  end859860861  # Process each comment and print the results862  USER_COMMENTS.each do |comment|863    puts "\nComment: #{comment}"864    violation, violated_categories, explanation = moderate_message(comment, UNSAFE_CATEGORIES)865866    if violation867      puts "Violated Categories: #{violated_categories.join(", ")}"868      puts "Explanation: #{explanation}"869    else870      puts "No issues detected."871    end872  end873  ```874</CodeGroup>875876In this example, the `moderate_message` function contains an assessment prompt that includes the unsafe content categories and the message to evaluate. The prompt asks Claude to assess whether the message should be moderated, based on the unsafe categories you defined earlier.877878The model's assessment is then parsed to determine if there is a violation. If there is a violation, Claude also returns a list of violated categories and an explanation as to why the message is unsafe.879880### Evaluate your prompt881882Content moderation is a classification problem. Thus, you can use the same techniques outlined in the [classification cookbook](https://platform.claude.com/cookbook/capabilities-classification-guide) to determine the accuracy of your content moderation system.883884One additional consideration is that instead of treating content moderation as a binary classification problem, you may instead create multiple categories to represent various risk levels. Creating multiple risk levels allows you to adjust the aggressiveness of your moderation. For example, you might want to automatically block user queries that are deemed high risk, while users with many medium risk queries are flagged for human review.885886<CodeGroup exclude="shell">887  ```python Python888  def assess_risk_level(message, unsafe_categories):889      # Convert the unsafe categories into a string, with each category on a new line890      unsafe_category_str = "\n".join(unsafe_categories)891892      # Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions893      assessment_prompt = f"""894      Assess the risk level of the following message,895      based on the unsafe categories listed below.896897  Message:898  <message>{message}</message>899900  Unsafe Categories:901  <categories>902  {unsafe_category_str}903  </categories>904905  Assign a risk level based on your confidence that the user's message should be moderated906  based on the defined unsafe categories:907908  0 - No risk909  1 - Low risk910  2 - Medium risk911  3 - High risk912913  Respond with ONLY a JSON object, using the format below:914  {{915    "risk_level": <Numerical field denoting the risk level>,916    "categories": [Comma-separated list of violated categories],917    "explanation": <Optional. Only include if risk level is greater than 0>918  }}919  Do not include markdown formatting or code fences in your response."""920921      # Send the request to Claude for risk assessment922      response = client.messages.create(923          model="claude-haiku-4-5-20251001",  # Using the Haiku model for lower costs924          max_tokens=200,925          messages=[{"role": "user", "content": assessment_prompt}],926      )927928      # Parse the JSON response from Claude929      text_block = next(block for block in response.content if block.type == "text")930      assessment = json.loads(text_block.text)931932      # Extract the risk level, violated categories, and explanation from the assessment933      risk_level = assessment["risk_level"]934      violated_categories = assessment["categories"]935      explanation = assessment.get("explanation")936937      return risk_level, violated_categories, explanation938939940  # Process each comment and print the results941  for comment in user_comments:942      print(f"\nComment: {comment}")943      risk_level, violated_categories, explanation = assess_risk_level(944          comment, unsafe_categories945      )946947      print(f"Risk Level: {risk_level}")948      if violated_categories:949          print(f"Violated Categories: {', '.join(violated_categories)}")950      if explanation:951          print(f"Explanation: {explanation}")952  ```953954  ```typescript TypeScript955  // Shape of the JSON risk assessment Claude returns956  interface RiskAssessment {957    risk_level: number;958    categories: string[];959    explanation?: string;960  }961962  async function assessRiskLevel(963    message: string,964    unsafeCategories: string[]965  ): Promise<{ riskLevel: number; violatedCategories: string[]; explanation?: string }> {966    // Convert the unsafe categories into a string, with each category on a new line967    const unsafeCategoryStr = unsafeCategories.join("\n");968969    // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions970    const assessmentPrompt = `971      Assess the risk level of the following message,972      based on the unsafe categories listed below.973974  Message:975  <message>${message}</message>976977  Unsafe Categories:978  <categories>979  ${unsafeCategoryStr}980  </categories>981982  Assign a risk level based on your confidence that the user's message should be moderated983  based on the defined unsafe categories:984985  0 - No risk986  1 - Low risk987  2 - Medium risk988  3 - High risk989990  Respond with ONLY a JSON object, using the format below:991  {992    "risk_level": <Numerical field denoting the risk level>,993    "categories": [Comma-separated list of violated categories],994    "explanation": <Optional. Only include if risk level is greater than 0>995  }996  Do not include markdown formatting or code fences in your response.`;997998    // Send the request to Claude for risk assessment999    const response = await client.messages.create({1000      model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs1001      max_tokens: 200,1002      messages: [{ role: "user", content: assessmentPrompt }]1003    });10041005    // Parse the JSON response from Claude1006    const textBlock = response.content.find((block) => block.type === "text");1007    if (!textBlock) {1008      throw new Error("Expected a text block in the response");1009    }1010    const assessment: RiskAssessment = JSON.parse(textBlock.text);10111012    // Extract the risk level, violated categories, and explanation from the assessment1013    const { risk_level: riskLevel, categories: violatedCategories, explanation } = assessment;10141015    return { riskLevel, violatedCategories, explanation };1016  }10171018  // Process each comment and print the results1019  for (const comment of userComments) {1020    console.log(`\nComment: ${comment}`);1021    const { riskLevel, violatedCategories, explanation } = await assessRiskLevel(1022      comment,1023      unsafeCategories1024    );10251026    console.log(`Risk Level: ${riskLevel}`);1027    if (violatedCategories.length > 0) {1028      console.log(`Violated Categories: ${violatedCategories.join(", ")}`);1029    }1030    if (explanation) {1031      console.log(`Explanation: ${explanation}`);1032    }1033  }1034  ```10351036  ```csharp C#1037  async Task<(int RiskLevel, List<string> ViolatedCategories, string? Explanation)> AssessRiskLevel(1038      string message,1039      IReadOnlyList<string> categories1040  )1041  {1042      // Convert the unsafe categories into a string, with each category on a new line1043      var unsafeCategoryText = string.Join("\n", categories);10441045      // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1046      var assessmentPrompt = $$"""10471048      Assess the risk level of the following message,1049      based on the unsafe categories listed below.10501051  Message:1052  <message>{{message}}</message>10531054  Unsafe Categories:1055  <categories>1056  {{unsafeCategoryText}}1057  </categories>10581059  Assign a risk level based on your confidence that the user's message should be moderated1060  based on the defined unsafe categories:10611062  0 - No risk1063  1 - Low risk1064  2 - Medium risk1065  3 - High risk10661067  Respond with ONLY a JSON object, using the format below:1068  {1069    "risk_level": <Numerical field denoting the risk level>,1070    "categories": [Comma-separated list of violated categories],1071    "explanation": <Optional. Only include if risk level is greater than 0>1072  }1073  Do not include markdown formatting or code fences in your response.1074  """;10751076      // Send the request to Claude for risk assessment1077      var response = await client.Messages.Create(1078          new()1079          {1080              Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1081              MaxTokens = 200,1082              Messages = [new() { Role = Role.User, Content = assessmentPrompt }],1083          }1084      );10851086      // Narrow the first content block to a text block, then parse Claude's JSON response1087      if (!response.Content[0].TryPickText(out var textBlock))1088      {1089          throw new InvalidOperationException("Expected a text response from Claude.");1090      }1091      var assessment = JsonNode.Parse(textBlock.Text)!;10921093      // Extract the risk level, violated categories, and explanation from the assessment1094      var riskLevel = assessment["risk_level"]!.GetValue<int>();1095      var violatedCategories = assessment["categories"]!1096          .AsArray()1097          .Select(category => category!.GetValue<string>())1098          .ToList();1099      var explanation = assessment["explanation"]?.GetValue<string>();11001101      return (riskLevel, violatedCategories, explanation);1102  }11031104  // Process each comment and print the results1105  foreach (var comment in userComments)1106  {1107      Console.WriteLine($"\nComment: {comment}");1108      var (riskLevel, violatedCategories, explanation) = await AssessRiskLevel(comment, unsafeCategories);11091110      Console.WriteLine($"Risk Level: {riskLevel}");1111      if (violatedCategories.Count > 0)1112      {1113          Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");1114      }1115      if (!string.IsNullOrEmpty(explanation))1116      {1117          Console.WriteLine($"Explanation: {explanation}");1118      }1119  }1120  ```11211122  ```go Go1123  func assessRiskLevel(message string, unsafeCategories []string) (int, []string, string) {1124  	// Convert the unsafe categories into a string, with each category on a new line1125  	unsafeCategoryStr := strings.Join(unsafeCategories, "\n")11261127  	// Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1128  	assessmentPrompt := fmt.Sprintf(`1129      Assess the risk level of the following message,1130      based on the unsafe categories listed below.11311132  Message:1133  <message>%s</message>11341135  Unsafe Categories:1136  <categories>1137  %s1138  </categories>11391140  Assign a risk level based on your confidence that the user's message should be moderated1141  based on the defined unsafe categories:11421143  0 - No risk1144  1 - Low risk1145  2 - Medium risk1146  3 - High risk11471148  Respond with ONLY a JSON object, using the format below:1149  {1150    "risk_level": <Numerical field denoting the risk level>,1151    "categories": [Comma-separated list of violated categories],1152    "explanation": <Optional. Only include if risk level is greater than 0>1153  }1154  Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)11551156  	// Send the request to Claude for risk assessment1157  	response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{1158  		Model:     anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1159  		MaxTokens: 200,1160  		Messages: []anthropic.MessageParam{1161  			anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),1162  		},1163  	})1164  	if err != nil {1165  		log.Fatal(err)1166  	}11671168  	// Narrow the first content block to a text block before reading its text1169  	textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)1170  	if !ok {1171  		log.Fatalf("expected a text block, got %q", response.Content[0].Type)1172  	}11731174  	// Parse the JSON response from Claude1175  	var assessment struct {1176  		RiskLevel   int      `json:"risk_level"`1177  		Categories  []string `json:"categories"`1178  		Explanation string   `json:"explanation"`1179  	}1180  	if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {1181  		log.Fatal(err)1182  	}11831184  	// Return the risk level, violated categories, and explanation from the assessment1185  	return assessment.RiskLevel, assessment.Categories, assessment.Explanation1186  }11871188  // assessAllRiskLevels processes each comment and prints the results.1189  func assessAllRiskLevels() {1190  	for _, comment := range userComments {1191  		fmt.Printf("\nComment: %s\n", comment)1192  		riskLevel, violatedCategories, explanation := assessRiskLevel(comment, unsafeCategories)11931194  		fmt.Printf("Risk Level: %d\n", riskLevel)1195  		if len(violatedCategories) > 0 {1196  			fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))1197  		}1198  		if explanation != "" {1199  			fmt.Printf("Explanation: %s\n", explanation)1200  		}1201  	}1202  }12031204  ```12051206  ```java Java1207  record RiskAssessment(int riskLevel, List<String> violatedCategories, String explanation) {}12081209  RiskAssessment assessRiskLevel(String message, List<String> unsafeCategories)1210          throws JsonProcessingException {1211      // Convert the unsafe categories into a string, with each category on a new line1212      String unsafeCategoryStr = String.join("\n", unsafeCategories);12131214      // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1215      String assessmentPrompt = """12161217              Assess the risk level of the following message,1218              based on the unsafe categories listed below.12191220          Message:1221          <message>%s</message>12221223          Unsafe Categories:1224          <categories>1225          %s1226          </categories>12271228          Assign a risk level based on your confidence that the user's message should be moderated1229          based on the defined unsafe categories:12301231          0 - No risk1232          1 - Low risk1233          2 - Medium risk1234          3 - High risk12351236          Respond with ONLY a JSON object, using the format below:1237          {1238            "risk_level": <Numerical field denoting the risk level>,1239            "categories": [Comma-separated list of violated categories],1240            "explanation": <Optional. Only include if risk level is greater than 0>1241          }1242          Do not include markdown formatting or code fences in your response."""1243              .formatted(message, unsafeCategoryStr);12441245      // Send the request to Claude for risk assessment1246      Message response = client.messages().create(MessageCreateParams.builder()1247              .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs1248              .maxTokens(200)1249              .addUserMessage(assessmentPrompt)1250              .build());12511252      // Parse the JSON response from Claude1253      String assessmentJson = response.content().stream()1254              .flatMap(contentBlock -> contentBlock.text().stream())1255              .findFirst()1256              .orElseThrow()1257              .text();1258      ObjectMapper mapper = new ObjectMapper();1259      JsonNode assessment = mapper.readTree(assessmentJson);12601261      // Extract the risk level, violated categories, and explanation from the assessment1262      int riskLevel = assessment.required("risk_level").asInt();1263      JsonNode categoriesNode = assessment.required("categories");1264      List<String> violatedCategories = categoriesNode.isNull()1265              ? List.of()1266              : mapper.convertValue(categoriesNode, new TypeReference<List<String>>() {});1267      String explanation = assessment.hasNonNull("explanation")1268              ? assessment.get("explanation").asText()1269              : null;12701271      return new RiskAssessment(riskLevel, violatedCategories, explanation);1272  }12731274  // Process each comment and print the results1275  void printRiskLevels() throws JsonProcessingException {1276      for (String comment : userComments) {1277          IO.println("\nComment: " + comment);1278          RiskAssessment assessment = assessRiskLevel(comment, unsafeCategories);12791280          IO.println("Risk Level: " + assessment.riskLevel());1281          if (!assessment.violatedCategories().isEmpty()) {1282              IO.println("Violated Categories: " + String.join(", ", assessment.violatedCategories()));1283          }1284          if (assessment.explanation() != null && !assessment.explanation().isEmpty()) {1285              IO.println("Explanation: " + assessment.explanation());1286          }1287      }1288  }1289  ```12901291  ```php PHP1292  $assessRiskLevel = function (string $message, array $unsafeCategories) use ($client): array {1293      // Convert the unsafe categories into a string, with each category on a new line1294      $unsafeCategoryStr = implode("\n", $unsafeCategories);12951296      // Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1297      $assessmentPrompt = <<<PROMPT12981299          Assess the risk level of the following message,1300          based on the unsafe categories listed below.13011302      Message:1303      <message>{$message}</message>13041305      Unsafe Categories:1306      <categories>1307      {$unsafeCategoryStr}1308      </categories>13091310      Assign a risk level based on your confidence that the user's message should be moderated1311      based on the defined unsafe categories:13121313      0 - No risk1314      1 - Low risk1315      2 - Medium risk1316      3 - High risk13171318      Respond with ONLY a JSON object, using the format below:1319      {1320        "risk_level": <Numerical field denoting the risk level>,1321        "categories": [Comma-separated list of violated categories],1322        "explanation": <Optional. Only include if risk level is greater than 0>1323      }1324      Do not include markdown formatting or code fences in your response.1325      PROMPT;13261327      // Send the request to Claude for risk assessment1328      $response = $client->messages->create(1329          model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs1330          maxTokens: 200,1331          messages: [['role' => 'user', 'content' => $assessmentPrompt]],1332      );13331334      // Parse the JSON response from Claude. The SDK decodes each content block1335      // into its concrete class, so find the TextBlock before reading the text.1336      $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)1337          ?? throw new RuntimeException('Expected a text block in the response.');1338      $assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);13391340      // Extract the risk level, violated categories, and explanation from the assessment1341      $riskLevel = $assessment['risk_level'];1342      $violatedCategories = $assessment['categories'];1343      $explanation = $assessment['explanation'] ?? null;13441345      return [$riskLevel, $violatedCategories, $explanation];1346  };13471348  // Process each comment and print the results1349  foreach ($userComments as $comment) {1350      echo "\nComment: {$comment}\n";1351      [$riskLevel, $violatedCategories, $explanation] = $assessRiskLevel($comment, $unsafeCategories);13521353      echo "Risk Level: {$riskLevel}\n";1354      if ($violatedCategories) {1355          echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";1356      }1357      if ($explanation) {1358          echo "Explanation: {$explanation}\n";1359      }1360  }1361  ```13621363  ```ruby Ruby1364  def assess_risk_level(message, unsafe_categories)1365    # Convert the unsafe categories into a string, with each category on a new line1366    unsafe_category_str = unsafe_categories.join("\n")13671368    # Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions1369    assessment_prompt = <<~PROMPT.chomp13701371          Assess the risk level of the following message,1372          based on the unsafe categories listed below.13731374      Message:1375      <message>#{message}</message>13761377      Unsafe Categories:1378      <categories>1379      #{unsafe_category_str}1380      </categories>13811382      Assign a risk level based on your confidence that the user's message should be moderated1383      based on the defined unsafe categories:13841385      0 - No risk1386      1 - Low risk1387      2 - Medium risk1388      3 - High risk13891390      Respond with ONLY a JSON object, using the format below:1391      {1392        "risk_level": <Numerical field denoting the risk level>,1393        "categories": [Comma-separated list of violated categories],1394        "explanation": <Optional. Only include if risk level is greater than 0>1395      }1396      Do not include markdown formatting or code fences in your response.1397    PROMPT13981399    # Send the request to Claude for risk assessment1400    response = CLIENT.messages.create(1401      model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs1402      max_tokens: 200,1403      messages: [{role: :user, content: assessment_prompt}]1404    )14051406    # Parse the JSON response from Claude1407    text_block = response.content.find { it.type == :text }1408    assessment = JSON.parse(text_block.text)14091410    # Extract the risk level, violated categories, and explanation from the assessment1411    risk_level = assessment["risk_level"]1412    violated_categories = assessment["categories"]1413    explanation = assessment["explanation"]14141415    [risk_level, violated_categories, explanation]1416  end141714181419  # Process each comment and print the results1420  USER_COMMENTS.each do |comment|1421    puts "\nComment: #{comment}"1422    risk_level, violated_categories, explanation = assess_risk_level(comment, UNSAFE_CATEGORIES)14231424    puts "Risk Level: #{risk_level}"1425    puts "Violated Categories: #{violated_categories.join(", ")}" if violated_categories&.any?1426    puts "Explanation: #{explanation}" if explanation1427  end1428  ```1429</CodeGroup>14301431This code implements an `assess_risk_level` function that uses Claude to evaluate the risk level of a message. The function accepts a message and the unsafe categories as inputs.14321433Within the function, a prompt is generated for Claude, including the message to be assessed, the unsafe categories, and specific instructions for evaluating the risk level. The prompt instructs Claude to respond with a JSON object that includes the risk level, the violated categories, and an optional explanation.14341435This approach enables flexible content moderation by assigning risk levels. It can be seamlessly integrated into a larger system to automate content filtering or flag comments for human review based on their assessed risk level. For instance, when running this code, the comment `Delete this post now or you better hide. I am coming after you and your family.` is identified as high risk because of its dangerous threat. Conversely, the comment `Stay away from the 5G cellphones!! They are using 5G to control you.` is categorized as medium risk.14361437### Deploy your prompt14381439Once you are confident in the quality of your solution, it's time to deploy it to production. Here are some best practices to follow when using content moderation in production:144014411. **Provide clear feedback to users:** When user input is blocked or a response is flagged because of content moderation, provide informative and constructive feedback to help users understand why their message was flagged and how they can rephrase it appropriately. In the earlier coding examples, this is done through the `explanation` field in the Claude response.144214432. **Analyze moderated content:** Keep track of the types of content being flagged by your moderation system to identify trends and potential areas for improvement.144414453. **Continuously evaluate and improve:** Regularly assess the performance of your content moderation system using metrics such as precision and recall tracking. Use this data to iteratively refine your moderation prompts, keywords, and assessment criteria.14461447***14481449## Improve performance14501451In complex scenarios, it may be helpful to consider additional strategies to improve performance beyond standard [prompt engineering techniques](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview). Here are some advanced strategies:14521453### Define topics and provide examples14541455In addition to listing the unsafe categories in the prompt, further improvements can be made by providing definitions and phrases related to each category.14561457<CodeGroup exclude="shell">1458  ```python Python1459  # Categories considered unsafe for content moderation, with their definitions1460  unsafe_category_definitions = {1461      "Child Exploitation": "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",1462      "Conspiracy Theories": """Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1463          which can incite harm or undermine public trust in institutions or facts.""",1464      "Hate": """Content that is hateful toward people on the basis of their protected characteristics1465          (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1466          or content that perpetuates negative stereotypes.""",1467      "Indiscriminate Weapons": "Content that enables the creation of weapons of mass destruction.",1468      "Intellectual Property": "Content that violates the intellectual property rights of third parties.",1469      "Non-Violent Crimes": "Content that enables, encourages, or excuses the commission of non-violent crimes.",1470      "Privacy": "Content that contains sensitive, personal information about private individuals.",1471      "Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",1472      "Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",1473      "Sexual Content": "Content that contains erotic descriptions or explicit depictions of sex acts.",1474      "Specialized Advice": """Content that contains financial, medical, or legal advice.1475          Financial advice includes guidance on investments, stocks, bonds, or any financial planning.""",1476      "Violent Crimes": "Content that enables, encourages, or excuses the commission of violent crimes.",1477  }147814791480  def moderate_message_with_definitions(message, unsafe_category_definitions):1481      # Format unsafe categories string, with each category and its definition on a new line1482      unsafe_category_str = "\n".join(1483          f"{category}: {definition}"1484          for category, definition in unsafe_category_definitions.items()1485      )14861487      # Construct the prompt for Claude, including the message and unsafe categories1488      assessment_prompt = f"""Determine whether the following message warrants moderation, based on the unsafe categories outlined below.14891490  Message:1491  <message>{message}</message>14921493  Unsafe Categories and Their Definitions:1494  <categories>1495  {unsafe_category_str}1496  </categories>14971498  It's important that you remember all unsafe categories and their definitions.14991500  Respond with ONLY a JSON object, using the format below:1501  {{1502    "violation": <Boolean field denoting whether the message should be moderated>,1503    "categories": [Comma-separated list of violated categories],1504    "explanation": [Optional. Only include if there is a violation.]1505  }}1506  Do not include markdown formatting or code fences in your response."""15071508      # Send the request to Claude for content moderation1509      response = client.messages.create(1510          model="claude-haiku-4-5-20251001",  # Using the Haiku model for lower costs1511          max_tokens=200,1512          messages=[{"role": "user", "content": assessment_prompt}],1513      )15141515      # Parse the JSON response from Claude1516      text_block = next(block for block in response.content if block.type == "text")1517      assessment = json.loads(text_block.text)15181519      # Extract the violation status from the assessment1520      contains_violation = assessment["violation"]15211522      # If there's a violation, get the categories and explanation; otherwise, use empty defaults1523      violated_categories = assessment.get("categories", []) if contains_violation else []1524      explanation = assessment.get("explanation") if contains_violation else None15251526      return contains_violation, violated_categories, explanation152715281529  # Process each comment and print the results1530  for comment in user_comments:1531      print(f"\nComment: {comment}")1532      violation, violated_categories, explanation = moderate_message_with_definitions(1533          comment, unsafe_category_definitions1534      )15351536      if violation:1537          print(f"Violated Categories: {', '.join(violated_categories)}")1538          print(f"Explanation: {explanation}")1539      else:1540          print("No issues detected.")1541  ```15421543  ```typescript TypeScript1544  // Shape of the JSON assessment Claude returns1545  interface DefinitionBasedAssessment {1546    violation: boolean;1547    categories?: string[];1548    explanation?: string;1549  }15501551  // Categories considered unsafe for content moderation, with their definitions1552  // (object keys preserve insertion order, so categories render in this order)1553  const unsafeCategoryDefinitions: Record<string, string> = {1554    "Child Exploitation":1555      "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",1556    "Conspiracy Theories": `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1557          which can incite harm or undermine public trust in institutions or facts.`,1558    "Hate": `Content that is hateful toward people on the basis of their protected characteristics1559          (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1560          or content that perpetuates negative stereotypes.`,1561    "Indiscriminate Weapons":1562      "Content that enables the creation of weapons of mass destruction.",1563    "Intellectual Property":1564      "Content that violates the intellectual property rights of third parties.",1565    "Non-Violent Crimes":1566      "Content that enables, encourages, or excuses the commission of non-violent crimes.",1567    "Privacy":1568      "Content that contains sensitive, personal information about private individuals.",1569    "Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",1570    "Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",1571    "Sexual Content":1572      "Content that contains erotic descriptions or explicit depictions of sex acts.",1573    "Specialized Advice": `Content that contains financial, medical, or legal advice.1574          Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`,1575    "Violent Crimes":1576      "Content that enables, encourages, or excuses the commission of violent crimes."1577  };15781579  async function moderateMessageWithDefinitions(1580    message: string,1581    unsafeCategoryDefinitions: Record<string, string>1582  ): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {1583    // Format the unsafe categories string, with each category and its definition on a new line1584    const unsafeCategoryStr = Object.entries(unsafeCategoryDefinitions)1585      .map(([category, definition]) => `${category}: ${definition}`)1586      .join("\n");15871588    // Construct the prompt for Claude, including the message and unsafe categories1589    const assessmentPrompt = `Determine whether the following message warrants moderation, based on the unsafe categories outlined below.15901591  Message:1592  <message>${message}</message>15931594  Unsafe Categories and Their Definitions:1595  <categories>1596  ${unsafeCategoryStr}1597  </categories>15981599  It's important that you remember all unsafe categories and their definitions.16001601  Respond with ONLY a JSON object, using the format below:1602  {1603    "violation": <Boolean field denoting whether the message should be moderated>,1604    "categories": [Comma-separated list of violated categories],1605    "explanation": [Optional. Only include if there is a violation.]1606  }1607  Do not include markdown formatting or code fences in your response.`;16081609    // Send the request to Claude for content moderation1610    const response = await client.messages.create({1611      model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs1612      max_tokens: 200,1613      messages: [{ role: "user", content: assessmentPrompt }]1614    });16151616    // Parse the JSON response from Claude1617    const textBlock = response.content.find((block) => block.type === "text");1618    if (!textBlock) {1619      throw new Error("Expected a text block in the response");1620    }1621    const assessment: DefinitionBasedAssessment = JSON.parse(textBlock.text);16221623    // Extract the violation status from the assessment1624    const containsViolation = assessment.violation;16251626    // If there's a violation, get the categories and explanation; otherwise, use empty defaults1627    const violatedCategories = containsViolation ? assessment.categories ?? [] : [];1628    const explanation = containsViolation ? assessment.explanation : undefined;16291630    return { violation: containsViolation, violatedCategories, explanation };1631  }16321633  // Process each comment and print the results1634  for (const comment of userComments) {1635    console.log(`\nComment: ${comment}`);1636    const { violation, violatedCategories, explanation } = await moderateMessageWithDefinitions(1637      comment,1638      unsafeCategoryDefinitions1639    );16401641    if (violation) {1642      console.log(`Violated Categories: ${violatedCategories.join(", ")}`);1643      console.log(`Explanation: ${explanation}`);1644    } else {1645      console.log("No issues detected.");1646    }1647  }1648  ```16491650  ```csharp C#1651  // Categories considered unsafe for content moderation, with their definitions.1652  // The entries stay in insertion order, so the rendered prompt lists categories1653  // in exactly this order.1654  (string Category, string Definition)[] unsafeCategoryDefinitions =1655  [1656      (1657          "Child Exploitation",1658          "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."1659      ),1660      (1661          "Conspiracy Theories",1662          """1663          Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1664                  which can incite harm or undermine public trust in institutions or facts.1665          """1666      ),1667      (1668          "Hate",1669          """1670          Content that is hateful toward people on the basis of their protected characteristics1671                  (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1672                  or content that perpetuates negative stereotypes.1673          """1674      ),1675      ("Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."),1676      ("Intellectual Property", "Content that violates the intellectual property rights of third parties."),1677      ("Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."),1678      ("Privacy", "Content that contains sensitive, personal information about private individuals."),1679      ("Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."),1680      ("Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."),1681      ("Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."),1682      (1683          "Specialized Advice",1684          """1685          Content that contains financial, medical, or legal advice.1686                  Financial advice includes guidance on investments, stocks, bonds, or any financial planning.1687          """1688      ),1689      ("Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."),1690  ];169116921693  async Task<(bool ContainsViolation, List<string> ViolatedCategories, string? Explanation)> ModerateMessageWithDefinitions(1694      string message,1695      IReadOnlyList<(string Category, string Definition)> categoryDefinitions1696  )1697  {1698      // Format the unsafe categories string, with each category and its definition on a new line1699      var unsafeCategoryText = string.Join(1700          "\n",1701          categoryDefinitions.Select(entry => $"{entry.Category}: {entry.Definition}")1702      );17031704      // Construct the prompt for Claude, including the message and unsafe categories1705      var assessmentPrompt = $$"""1706  Determine whether the following message warrants moderation, based on the unsafe categories outlined below.17071708  Message:1709  <message>{{message}}</message>17101711  Unsafe Categories and Their Definitions:1712  <categories>1713  {{unsafeCategoryText}}1714  </categories>17151716  It's important that you remember all unsafe categories and their definitions.17171718  Respond with ONLY a JSON object, using the format below:1719  {1720    "violation": <Boolean field denoting whether the message should be moderated>,1721    "categories": [Comma-separated list of violated categories],1722    "explanation": [Optional. Only include if there is a violation.]1723  }1724  Do not include markdown formatting or code fences in your response.1725  """;17261727      // Send the request to Claude for content moderation1728      var response = await client.Messages.Create(1729          new()1730          {1731              Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1732              MaxTokens = 200,1733              Messages = [new() { Role = Role.User, Content = assessmentPrompt }],1734          }1735      );17361737      // Narrow the first content block to a text block, then parse Claude's JSON response1738      if (!response.Content[0].TryPickText(out var textBlock))1739      {1740          throw new InvalidOperationException("Expected a text response from Claude.");1741      }1742      var assessment = JsonNode.Parse(textBlock.Text)!;17431744      // Extract the violation status from the assessment1745      var containsViolation = assessment["violation"]!.GetValue<bool>();17461747      // If there's a violation, get the categories and explanation; otherwise, use empty defaults1748      List<string> violatedCategories = containsViolation1749          ? assessment["categories"]?.AsArray().Select(category => category!.GetValue<string>()).ToList() ?? []1750          : [];1751      var explanation = containsViolation ? assessment["explanation"]?.GetValue<string>() : null;17521753      return (containsViolation, violatedCategories, explanation);1754  }17551756  // Process each comment and print the results1757  foreach (var comment in userComments)1758  {1759      Console.WriteLine($"\nComment: {comment}");1760      var (violation, violatedCategories, explanation) = await ModerateMessageWithDefinitions(1761          comment,1762          unsafeCategoryDefinitions1763      );17641765      if (violation)1766      {1767          Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");1768          Console.WriteLine($"Explanation: {explanation}");1769      }1770      else1771      {1772          Console.WriteLine("No issues detected.");1773      }1774  }1775  ```17761777  ```go Go1778  // Categories considered unsafe for content moderation, with their definitions.1779  // A slice of category/definition pairs (rather than a map) keeps the rendered1780  // order stable; Go maps iterate in random order.1781  type categoryDefinition struct {1782  	category   string1783  	definition string1784  }17851786  var unsafeCategoryDefinitions = []categoryDefinition{1787  	{"Child Exploitation", "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."},1788  	{"Conspiracy Theories", `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1789          which can incite harm or undermine public trust in institutions or facts.`},1790  	{"Hate", `Content that is hateful toward people on the basis of their protected characteristics1791          (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1792          or content that perpetuates negative stereotypes.`},1793  	{"Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."},1794  	{"Intellectual Property", "Content that violates the intellectual property rights of third parties."},1795  	{"Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."},1796  	{"Privacy", "Content that contains sensitive, personal information about private individuals."},1797  	{"Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."},1798  	{"Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."},1799  	{"Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."},1800  	{"Specialized Advice", `Content that contains financial, medical, or legal advice.1801          Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`},1802  	{"Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."},1803  }18041805  func moderateMessageWithDefinitions(message string, unsafeCategoryDefinitions []categoryDefinition) (bool, []string, string) {1806  	// Format unsafe categories string, with each category and its definition on a new line1807  	categoryLines := make([]string, len(unsafeCategoryDefinitions))1808  	for i, entry := range unsafeCategoryDefinitions {1809  		categoryLines[i] = fmt.Sprintf("%s: %s", entry.category, entry.definition)1810  	}1811  	unsafeCategoryStr := strings.Join(categoryLines, "\n")18121813  	// Construct the prompt for Claude, including the message and unsafe categories1814  	assessmentPrompt := fmt.Sprintf(`Determine whether the following message warrants moderation, based on the unsafe categories outlined below.18151816  Message:1817  <message>%s</message>18181819  Unsafe Categories and Their Definitions:1820  <categories>1821  %s1822  </categories>18231824  It's important that you remember all unsafe categories and their definitions.18251826  Respond with ONLY a JSON object, using the format below:1827  {1828    "violation": <Boolean field denoting whether the message should be moderated>,1829    "categories": [Comma-separated list of violated categories],1830    "explanation": [Optional. Only include if there is a violation.]1831  }1832  Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)18331834  	// Send the request to Claude for content moderation1835  	response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{1836  		Model:     anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs1837  		MaxTokens: 200,1838  		Messages: []anthropic.MessageParam{1839  			anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),1840  		},1841  	})1842  	if err != nil {1843  		log.Fatal(err)1844  	}18451846  	// Narrow the first content block to a text block before reading its text1847  	textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)1848  	if !ok {1849  		log.Fatalf("expected a text block, got %q", response.Content[0].Type)1850  	}18511852  	// Parse the JSON response from Claude1853  	var assessment struct {1854  		Violation   bool     `json:"violation"`1855  		Categories  []string `json:"categories"`1856  		Explanation string   `json:"explanation"`1857  	}1858  	if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {1859  		log.Fatal(err)1860  	}18611862  	// If there's a violation, return the categories and explanation; otherwise, use empty defaults1863  	if !assessment.Violation {1864  		return false, nil, ""1865  	}1866  	return true, assessment.Categories, assessment.Explanation1867  }18681869  // moderateAllCommentsWithDefinitions processes each comment and prints the results.1870  func moderateAllCommentsWithDefinitions() {1871  	for _, comment := range userComments {1872  		fmt.Printf("\nComment: %s\n", comment)1873  		violation, violatedCategories, explanation := moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions)18741875  		if violation {1876  			fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))1877  			fmt.Printf("Explanation: %s\n", explanation)1878  		} else {1879  			fmt.Println("No issues detected.")1880  		}1881  	}1882  }18831884  ```18851886  ```java Java1887  // Categories considered unsafe for content moderation, with their definitions1888  record CategoryDefinition(String category, String definition) {}18891890  final List<CategoryDefinition> unsafeCategoryDefinitions = List.of(1891          new CategoryDefinition(1892                  "Child Exploitation",1893                  "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."),1894          new CategoryDefinition(1895                  "Conspiracy Theories",1896                  """1897                  Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,1898                          which can incite harm or undermine public trust in institutions or facts."""),1899          new CategoryDefinition(1900                  "Hate",1901                  """1902                  Content that is hateful toward people on the basis of their protected characteristics1903                          (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),1904                          or content that perpetuates negative stereotypes."""),1905          new CategoryDefinition(1906                  "Indiscriminate Weapons",1907                  "Content that enables the creation of weapons of mass destruction."),1908          new CategoryDefinition(1909                  "Intellectual Property",1910                  "Content that violates the intellectual property rights of third parties."),1911          new CategoryDefinition(1912                  "Non-Violent Crimes",1913                  "Content that enables, encourages, or excuses the commission of non-violent crimes."),1914          new CategoryDefinition(1915                  "Privacy",1916                  "Content that contains sensitive, personal information about private individuals."),1917          new CategoryDefinition(1918                  "Self-Harm",1919                  "Content that enables, encourages, or excuses acts of intentional self-harm."),1920          new CategoryDefinition(1921                  "Sex Crimes",1922                  "Content that enables, encourages, or excuses the commission of sex crimes."),1923          new CategoryDefinition(1924                  "Sexual Content",1925                  "Content that contains erotic descriptions or explicit depictions of sex acts."),1926          new CategoryDefinition(1927                  "Specialized Advice",1928                  """1929                  Content that contains financial, medical, or legal advice.1930                          Financial advice includes guidance on investments, stocks, bonds, or any financial planning."""),1931          new CategoryDefinition(1932                  "Violent Crimes",1933                  "Content that enables, encourages, or excuses the commission of violent crimes."));19341935  record ModerationDecision(boolean violation, List<String> violatedCategories, String explanation) {}19361937  ModerationDecision moderateMessageWithDefinitions(1938          String message, List<CategoryDefinition> unsafeCategoryDefinitions)1939          throws JsonProcessingException {1940      // Format unsafe categories string, with each category and its definition on a new line1941      String unsafeCategoryStr = unsafeCategoryDefinitions.stream()1942              .map(categoryDefinition ->1943                      categoryDefinition.category() + ": " + categoryDefinition.definition())1944              .collect(Collectors.joining("\n"));19451946      // Construct the prompt for Claude, including the message and unsafe categories1947      String assessmentPrompt = """1948          Determine whether the following message warrants moderation, based on the unsafe categories outlined below.19491950          Message:1951          <message>%s</message>19521953          Unsafe Categories and Their Definitions:1954          <categories>1955          %s1956          </categories>19571958          It's important that you remember all unsafe categories and their definitions.19591960          Respond with ONLY a JSON object, using the format below:1961          {1962            "violation": <Boolean field denoting whether the message should be moderated>,1963            "categories": [Comma-separated list of violated categories],1964            "explanation": [Optional. Only include if there is a violation.]1965          }1966          Do not include markdown formatting or code fences in your response."""1967              .formatted(message, unsafeCategoryStr);19681969      // Send the request to Claude for content moderation1970      Message response = client.messages().create(MessageCreateParams.builder()1971              .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs1972              .maxTokens(200)1973              .addUserMessage(assessmentPrompt)1974              .build());19751976      // Parse the JSON response from Claude1977      String assessmentJson = response.content().stream()1978              .flatMap(contentBlock -> contentBlock.text().stream())1979              .findFirst()1980              .orElseThrow()1981              .text();1982      ObjectMapper mapper = new ObjectMapper();1983      JsonNode assessment = mapper.readTree(assessmentJson);19841985      // Extract the violation status from the assessment1986      boolean containsViolation = assessment.required("violation").asBoolean();19871988      // If there's a violation, get the categories and explanation; otherwise, use empty defaults1989      List<String> violatedCategories = containsViolation && assessment.has("categories")1990              ? mapper.convertValue(assessment.get("categories"), new TypeReference<List<String>>() {})1991              : List.of();1992      String explanation = containsViolation && assessment.hasNonNull("explanation")1993              ? assessment.get("explanation").asText()1994              : null;19951996      return new ModerationDecision(containsViolation, violatedCategories, explanation);1997  }19981999  // Process each comment and print the results2000  void printModerationResultsWithDefinitions() throws JsonProcessingException {2001      for (String comment : userComments) {2002          IO.println("\nComment: " + comment);2003          ModerationDecision result = moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions);20042005          if (result.violation()) {2006              IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));2007              IO.println("Explanation: " + result.explanation());2008          } else {2009              IO.println("No issues detected.");2010          }2011      }2012  }2013  ```20142015  ```php PHP2016  // Categories considered unsafe for content moderation, with their definitions2017  $unsafeCategoryDefinitions = [2018      'Child Exploitation' => 'Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.',2019      'Conspiracy Theories' => 'Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,2020          which can incite harm or undermine public trust in institutions or facts.',2021      'Hate' => 'Content that is hateful toward people on the basis of their protected characteristics2022          (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),2023          or content that perpetuates negative stereotypes.',2024      'Indiscriminate Weapons' => 'Content that enables the creation of weapons of mass destruction.',2025      'Intellectual Property' => 'Content that violates the intellectual property rights of third parties.',2026      'Non-Violent Crimes' => 'Content that enables, encourages, or excuses the commission of non-violent crimes.',2027      'Privacy' => 'Content that contains sensitive, personal information about private individuals.',2028      'Self-Harm' => 'Content that enables, encourages, or excuses acts of intentional self-harm.',2029      'Sex Crimes' => 'Content that enables, encourages, or excuses the commission of sex crimes.',2030      'Sexual Content' => 'Content that contains erotic descriptions or explicit depictions of sex acts.',2031      'Specialized Advice' => 'Content that contains financial, medical, or legal advice.2032          Financial advice includes guidance on investments, stocks, bonds, or any financial planning.',2033      'Violent Crimes' => 'Content that enables, encourages, or excuses the commission of violent crimes.',2034  ];20352036  $moderateMessageWithDefinitions = function (string $message, array $unsafeCategoryDefinitions) use ($client): array {2037      // Format the unsafe categories string, with each category and its definition on a new line2038      $categoryLines = [];2039      foreach ($unsafeCategoryDefinitions as $category => $definition) {2040          $categoryLines[] = "{$category}: {$definition}";2041      }2042      $unsafeCategoryStr = implode("\n", $categoryLines);20432044      // Construct the prompt for Claude, including the message and unsafe categories2045      $assessmentPrompt = <<<PROMPT2046      Determine whether the following message warrants moderation, based on the unsafe categories outlined below.20472048      Message:2049      <message>{$message}</message>20502051      Unsafe Categories and Their Definitions:2052      <categories>2053      {$unsafeCategoryStr}2054      </categories>20552056      It's important that you remember all unsafe categories and their definitions.20572058      Respond with ONLY a JSON object, using the format below:2059      {2060        "violation": <Boolean field denoting whether the message should be moderated>,2061        "categories": [Comma-separated list of violated categories],2062        "explanation": [Optional. Only include if there is a violation.]2063      }2064      Do not include markdown formatting or code fences in your response.2065      PROMPT;20662067      // Send the request to Claude for content moderation2068      $response = $client->messages->create(2069          model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs2070          maxTokens: 200,2071          messages: [['role' => 'user', 'content' => $assessmentPrompt]],2072      );20732074      // Parse the JSON response from Claude. The SDK decodes each content block2075      // into its concrete class, so find the TextBlock before reading the text.2076      $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)2077          ?? throw new RuntimeException('Expected a text block in the response.');2078      $assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);20792080      // Extract the violation status from the assessment2081      $containsViolation = $assessment['violation'];20822083      // If there's a violation, get the categories and explanation; otherwise, use empty defaults2084      $violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];2085      $explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;20862087      return [$containsViolation, $violatedCategories, $explanation];2088  };20892090  // Process each comment and print the results2091  foreach ($userComments as $comment) {2092      echo "\nComment: {$comment}\n";2093      [$violation, $violatedCategories, $explanation] = $moderateMessageWithDefinitions($comment, $unsafeCategoryDefinitions);20942095      if ($violation) {2096          echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";2097          echo "Explanation: {$explanation}\n";2098      } else {2099          echo "No issues detected.\n";2100      }2101  }2102  ```21032104  ```ruby Ruby2105  # Categories considered unsafe for content moderation, with their definitions2106  UNSAFE_CATEGORY_DEFINITIONS = {2107    "Child Exploitation" => "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",2108    "Conspiracy Theories" => "Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,2109          which can incite harm or undermine public trust in institutions or facts.",2110    "Hate" => "Content that is hateful toward people on the basis of their protected characteristics2111          (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),2112          or content that perpetuates negative stereotypes.",2113    "Indiscriminate Weapons" => "Content that enables the creation of weapons of mass destruction.",2114    "Intellectual Property" => "Content that violates the intellectual property rights of third parties.",2115    "Non-Violent Crimes" => "Content that enables, encourages, or excuses the commission of non-violent crimes.",2116    "Privacy" => "Content that contains sensitive, personal information about private individuals.",2117    "Self-Harm" => "Content that enables, encourages, or excuses acts of intentional self-harm.",2118    "Sex Crimes" => "Content that enables, encourages, or excuses the commission of sex crimes.",2119    "Sexual Content" => "Content that contains erotic descriptions or explicit depictions of sex acts.",2120    "Specialized Advice" => "Content that contains financial, medical, or legal advice.2121          Financial advice includes guidance on investments, stocks, bonds, or any financial planning.",2122    "Violent Crimes" => "Content that enables, encourages, or excuses the commission of violent crimes."2123  }212421252126  def moderate_message_with_definitions(message, unsafe_category_definitions)2127    # Format the unsafe categories string, with each category and its definition on a new line2128    unsafe_category_str = unsafe_category_definitions2129      .map { |category, definition| "#{category}: #{definition}" }2130      .join("\n")21312132    # Construct the prompt for Claude, including the message and unsafe categories2133    assessment_prompt = <<~PROMPT.chomp2134      Determine whether the following message warrants moderation, based on the unsafe categories outlined below.21352136      Message:2137      <message>#{message}</message>21382139      Unsafe Categories and Their Definitions:2140      <categories>2141      #{unsafe_category_str}2142      </categories>21432144      It's important that you remember all unsafe categories and their definitions.21452146      Respond with ONLY a JSON object, using the format below:2147      {2148        "violation": <Boolean field denoting whether the message should be moderated>,2149        "categories": [Comma-separated list of violated categories],2150        "explanation": [Optional. Only include if there is a violation.]2151      }2152      Do not include markdown formatting or code fences in your response.2153    PROMPT21542155    # Send the request to Claude for content moderation2156    response = CLIENT.messages.create(2157      model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs2158      max_tokens: 200,2159      messages: [{role: :user, content: assessment_prompt}]2160    )21612162    # Parse the JSON response from Claude2163    text_block = response.content.find { it.type == :text }2164    assessment = JSON.parse(text_block.text)21652166    # Extract the violation status from the assessment2167    contains_violation = assessment["violation"]21682169    # If there's a violation, get the categories and explanation; otherwise, use empty defaults2170    violated_categories = contains_violation ? assessment.fetch("categories", []) : []2171    explanation = contains_violation ? assessment["explanation"] : nil21722173    [contains_violation, violated_categories, explanation]2174  end217521762177  # Process each comment and print the results2178  USER_COMMENTS.each do |comment|2179    puts "\nComment: #{comment}"2180    violation, violated_categories, explanation = moderate_message_with_definitions(comment, UNSAFE_CATEGORY_DEFINITIONS)21812182    if violation2183      puts "Violated Categories: #{violated_categories.join(", ")}"2184      puts "Explanation: #{explanation}"2185    else2186      puts "No issues detected."2187    end2188  end2189  ```2190</CodeGroup>21912192The `moderate_message_with_definitions` function expands upon the earlier `moderate_message` function by allowing each unsafe category to be paired with a detailed definition. This occurs in the code by replacing the `unsafe_categories` collection from the original function with an `unsafe_category_definitions` mapping. This mapping pairs each unsafe category with its corresponding definition. Both the category names and their definitions are included in the prompt.21932194Notably, the definition for the `Specialized Advice` category now specifies the types of financial advice that should be prohibited. As a result, the comment `It is a great time to invest in gold!`, which previously passed the `moderate_message` assessment, now triggers a violation.21952196### Consider batch processing21972198To reduce costs in situations where real-time moderation isn't necessary, consider moderating messages in batches. Include multiple messages within the prompt's context, and ask Claude to assess which messages should be moderated.21992200<CodeGroup exclude="shell">2201  ```python Python2202  def batch_moderate_messages(messages, unsafe_categories):2203      # Convert the unsafe categories into a string, with each category on a new line2204      unsafe_category_str = "\n".join(unsafe_categories)22052206      # Format messages string, with each message wrapped in XML-like tags and given an ID2207      messages_str = "\n".join(2208          [f"<message id={idx}>{msg}</message>" for idx, msg in enumerate(messages)]2209      )22102211      # Construct the prompt for Claude, including the messages and unsafe categories2212      assessment_prompt = f"""Determine the messages to moderate, based on the unsafe categories outlined below.22132214  Messages:2215  <messages>2216  {messages_str}2217  </messages>22182219  Unsafe Categories:2220  <categories>2221  {unsafe_category_str}2222  </categories>22232224  Respond with ONLY a JSON object, using the format below:2225  {{2226    "violations": [2227      {{2228        "id": <message id>,2229        "categories": [list of violated categories],2230        "explanation": <Explanation of why there's a violation>2231      }}2232    ]2233  }}22342235  Important Notes:2236  - Remember to analyze every message for a violation.2237  - Select any number of violations that reasonably apply.2238  - Do not include markdown formatting or code fences in your response."""22392240      # Send the request to Claude for content moderation2241      response = client.messages.create(2242          model="claude-haiku-4-5-20251001",  # Using the Haiku model for lower costs2243          max_tokens=2048,  # Increased max token count to handle batches2244          messages=[{"role": "user", "content": assessment_prompt}],2245      )22462247      # Parse the JSON response from Claude2248      text_block = next(block for block in response.content if block.type == "text")2249      assessment = json.loads(text_block.text)2250      return assessment225122522253  # Process the batch of comments and get the response2254  response_obj = batch_moderate_messages(user_comments, unsafe_categories)22552256  # Print the results for each detected violation2257  for violation in response_obj["violations"]:2258      print(f"""Comment: {user_comments[violation["id"]]}2259  Violated Categories: {", ".join(violation["categories"])}2260  Explanation: {violation["explanation"]}2261  """)2262  ```22632264  ```typescript TypeScript2265  // Shape of the JSON batch assessment Claude returns2266  interface BatchAssessment {2267    violations: {2268      id: number;2269      categories: string[];2270      explanation: string;2271    }[];2272  }22732274  async function batchModerateMessages(2275    messages: string[],2276    unsafeCategories: string[]2277  ): Promise<BatchAssessment> {2278    // Convert the unsafe categories into a string, with each category on a new line2279    const unsafeCategoryStr = unsafeCategories.join("\n");22802281    // Format the messages string, with each message wrapped in XML-like tags and given an ID2282    const messagesStr = messages2283      .map((msg, idx) => `<message id=${idx}>${msg}</message>`)2284      .join("\n");22852286    // Construct the prompt for Claude, including the messages and unsafe categories2287    const assessmentPrompt = `Determine the messages to moderate, based on the unsafe categories outlined below.22882289  Messages:2290  <messages>2291  ${messagesStr}2292  </messages>22932294  Unsafe Categories:2295  <categories>2296  ${unsafeCategoryStr}2297  </categories>22982299  Respond with ONLY a JSON object, using the format below:2300  {2301    "violations": [2302      {2303        "id": <message id>,2304        "categories": [list of violated categories],2305        "explanation": <Explanation of why there's a violation>2306      }2307    ]2308  }23092310  Important Notes:2311  - Remember to analyze every message for a violation.2312  - Select any number of violations that reasonably apply.2313  - Do not include markdown formatting or code fences in your response.`;23142315    // Send the request to Claude for content moderation2316    const response = await client.messages.create({2317      model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs2318      max_tokens: 2048, // Increased max token count to handle batches2319      messages: [{ role: "user", content: assessmentPrompt }]2320    });23212322    // Parse the JSON response from Claude2323    const textBlock = response.content.find((block) => block.type === "text");2324    if (!textBlock) {2325      throw new Error("Expected a text block in the response");2326    }2327    const assessment: BatchAssessment = JSON.parse(textBlock.text);2328    return assessment;2329  }23302331  // Process the batch of comments and get the response2332  const batchAssessment = await batchModerateMessages(userComments, unsafeCategories);23332334  // Print the results for each detected violation2335  for (const violation of batchAssessment.violations) {2336    console.log(`Comment: ${userComments[violation.id]}2337  Violated Categories: ${violation.categories.join(", ")}2338  Explanation: ${violation.explanation}2339  `);2340  }2341  ```23422343  ```csharp C#2344  async Task<JsonNode> BatchModerateMessages(IReadOnlyList<string> messages, IReadOnlyList<string> categories)2345  {2346      // Convert the unsafe categories into a string, with each category on a new line2347      var unsafeCategoryText = string.Join("\n", categories);23482349      // Format the messages string, with each message wrapped in XML-like tags and given an ID2350      var messagesText = string.Join(2351          "\n",2352          messages.Select((message, index) => $"<message id={index}>{message}</message>")2353      );23542355      // Construct the prompt for Claude, including the messages and unsafe categories2356      var assessmentPrompt = $$"""2357  Determine the messages to moderate, based on the unsafe categories outlined below.23582359  Messages:2360  <messages>2361  {{messagesText}}2362  </messages>23632364  Unsafe Categories:2365  <categories>2366  {{unsafeCategoryText}}2367  </categories>23682369  Respond with ONLY a JSON object, using the format below:2370  {2371    "violations": [2372      {2373        "id": <message id>,2374        "categories": [list of violated categories],2375        "explanation": <Explanation of why there's a violation>2376      }2377    ]2378  }23792380  Important Notes:2381  - Remember to analyze every message for a violation.2382  - Select any number of violations that reasonably apply.2383  - Do not include markdown formatting or code fences in your response.2384  """;23852386      // Send the request to Claude for content moderation2387      var response = await client.Messages.Create(2388          new()2389          {2390              Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs2391              MaxTokens = 2048, // Increased max token count to handle batches2392              Messages = [new() { Role = Role.User, Content = assessmentPrompt }],2393          }2394      );23952396      // Narrow the first content block to a text block, then parse Claude's JSON response2397      if (!response.Content[0].TryPickText(out var textBlock))2398      {2399          throw new InvalidOperationException("Expected a text response from Claude.");2400      }2401      return JsonNode.Parse(textBlock.Text)!;2402  }24032404  // Process the batch of comments and get the response2405  var moderationResults = await BatchModerateMessages(userComments, unsafeCategories);24062407  // Print the results for each detected violation2408  foreach (var violation in moderationResults["violations"]!.AsArray())2409  {2410      var flaggedComment = userComments[violation!["id"]!.GetValue<int>()];2411      var violatedCategories = string.Join(2412          ", ",2413          violation["categories"]!.AsArray().Select(category => category!.GetValue<string>())2414      );2415      var explanation = violation["explanation"]!.GetValue<string>();24162417      Console.WriteLine($"""2418          Comment: {flaggedComment}2419          Violated Categories: {violatedCategories}2420          Explanation: {explanation}24212422          """);2423  }2424  ```24252426  ```go Go2427  // batchViolation is one entry in Claude's "violations" array: the index of the2428  // offending message plus the categories it violated and why.2429  type batchViolation struct {2430  	ID          int      `json:"id"`2431  	Categories  []string `json:"categories"`2432  	Explanation string   `json:"explanation"`2433  }24342435  func batchModerateMessages(messages []string, unsafeCategories []string) []batchViolation {2436  	// Convert the unsafe categories into a string, with each category on a new line2437  	unsafeCategoryStr := strings.Join(unsafeCategories, "\n")24382439  	// Format messages string, with each message wrapped in XML-like tags and given an ID2440  	messageLines := make([]string, len(messages))2441  	for i, message := range messages {2442  		messageLines[i] = fmt.Sprintf("<message id=%d>%s</message>", i, message)2443  	}2444  	messagesStr := strings.Join(messageLines, "\n")24452446  	// Construct the prompt for Claude, including the messages and unsafe categories2447  	assessmentPrompt := fmt.Sprintf(`Determine the messages to moderate, based on the unsafe categories outlined below.24482449  Messages:2450  <messages>2451  %s2452  </messages>24532454  Unsafe Categories:2455  <categories>2456  %s2457  </categories>24582459  Respond with ONLY a JSON object, using the format below:2460  {2461    "violations": [2462      {2463        "id": <message id>,2464        "categories": [list of violated categories],2465        "explanation": <Explanation of why there's a violation>2466      }2467    ]2468  }24692470  Important Notes:2471  - Remember to analyze every message for a violation.2472  - Select any number of violations that reasonably apply.2473  - Do not include markdown formatting or code fences in your response.`, messagesStr, unsafeCategoryStr)24742475  	// Send the request to Claude for content moderation2476  	response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{2477  		Model:     anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs2478  		MaxTokens: 2048,                                   // Increased max token count to handle batches2479  		Messages: []anthropic.MessageParam{2480  			anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),2481  		},2482  	})2483  	if err != nil {2484  		log.Fatal(err)2485  	}24862487  	// Narrow the first content block to a text block before reading its text2488  	textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)2489  	if !ok {2490  		log.Fatalf("expected a text block, got %q", response.Content[0].Type)2491  	}24922493  	// Parse the JSON response from Claude2494  	var assessment struct {2495  		Violations []batchViolation `json:"violations"`2496  	}2497  	if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {2498  		log.Fatal(err)2499  	}2500  	return assessment.Violations2501  }25022503  // moderateAllCommentsAsBatch moderates the whole batch of comments in a single2504  // request and prints the results for each detected violation.2505  func moderateAllCommentsAsBatch() {2506  	// Process the batch of comments and get the response2507  	violations := batchModerateMessages(userComments, unsafeCategories)25082509  	// Print the results for each detected violation2510  	for _, violation := range violations {2511  		fmt.Printf(`Comment: %s2512  Violated Categories: %s2513  Explanation: %s25142515  `, userComments[violation.ID], strings.Join(violation.Categories, ", "), violation.Explanation)2516  	}2517  }25182519  ```25202521  ```java Java2522  JsonNode batchModerateMessages(List<String> messages, List<String> unsafeCategories)2523          throws JsonProcessingException {2524      // Convert the unsafe categories into a string, with each category on a new line2525      String unsafeCategoryStr = String.join("\n", unsafeCategories);25262527      // Format messages string, with each message wrapped in XML-like tags and given an ID2528      String messagesStr = IntStream.range(0, messages.size())2529              .mapToObj(idx -> "<message id=%d>%s</message>".formatted(idx, messages.get(idx)))2530              .collect(Collectors.joining("\n"));25312532      // Construct the prompt for Claude, including the messages and unsafe categories2533      String assessmentPrompt = """2534          Determine the messages to moderate, based on the unsafe categories outlined below.25352536          Messages:2537          <messages>2538          %s2539          </messages>25402541          Unsafe Categories:2542          <categories>2543          %s2544          </categories>25452546          Respond with ONLY a JSON object, using the format below:2547          {2548            "violations": [2549              {2550                "id": <message id>,2551                "categories": [list of violated categories],2552                "explanation": <Explanation of why there's a violation>2553              }2554            ]2555          }25562557          Important Notes:2558          - Remember to analyze every message for a violation.2559          - Select any number of violations that reasonably apply.2560          - Do not include markdown formatting or code fences in your response."""2561              .formatted(messagesStr, unsafeCategoryStr);25622563      // Send the request to Claude for content moderation2564      Message response = client.messages().create(MessageCreateParams.builder()2565              .model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs2566              .maxTokens(2048) // Increased max token count to handle batches2567              .addUserMessage(assessmentPrompt)2568              .build());25692570      // Parse the JSON response from Claude2571      String assessmentJson = response.content().stream()2572              .flatMap(contentBlock -> contentBlock.text().stream())2573              .findFirst()2574              .orElseThrow()2575              .text();2576      return new ObjectMapper().readTree(assessmentJson);2577  }25782579  // Process the batch of comments and print the results for each detected violation2580  void printBatchViolations() throws JsonProcessingException {2581      JsonNode response = batchModerateMessages(userComments, unsafeCategories);25822583      ObjectMapper mapper = new ObjectMapper();2584      for (JsonNode violation : response.required("violations")) {2585          List<String> violatedCategories =2586                  mapper.convertValue(violation.required("categories"), new TypeReference<List<String>>() {});2587          IO.println("""2588                  Comment: %s2589                  Violated Categories: %s2590                  Explanation: %s2591                  """.formatted(2592                          userComments.get(violation.required("id").asInt()),2593                          String.join(", ", violatedCategories),2594                          violation.required("explanation").asText()));2595      }2596  }2597  ```25982599  ```php PHP2600  $batchModerateMessages = function (array $messages, array $unsafeCategories) use ($client): array {2601      // Convert the unsafe categories into a string, with each category on a new line2602      $unsafeCategoryStr = implode("\n", $unsafeCategories);26032604      // Format the messages string, with each message wrapped in XML-like tags and given an ID2605      $messageLines = [];2606      foreach ($messages as $idx => $msg) {2607          $messageLines[] = "<message id={$idx}>{$msg}</message>";2608      }2609      $messagesStr = implode("\n", $messageLines);26102611      // Construct the prompt for Claude, including the messages and unsafe categories2612      $assessmentPrompt = <<<PROMPT2613      Determine the messages to moderate, based on the unsafe categories outlined below.26142615      Messages:2616      <messages>2617      {$messagesStr}2618      </messages>26192620      Unsafe Categories:2621      <categories>2622      {$unsafeCategoryStr}2623      </categories>26242625      Respond with ONLY a JSON object, using the format below:2626      {2627        "violations": [2628          {2629            "id": <message id>,2630            "categories": [list of violated categories],2631            "explanation": <Explanation of why there's a violation>2632          }2633        ]2634      }26352636      Important Notes:2637      - Remember to analyze every message for a violation.2638      - Select any number of violations that reasonably apply.2639      - Do not include markdown formatting or code fences in your response.2640      PROMPT;26412642      // Send the request to Claude for content moderation2643      $response = $client->messages->create(2644          model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs2645          maxTokens: 2048, // Increased max token count to handle batches2646          messages: [['role' => 'user', 'content' => $assessmentPrompt]],2647      );26482649      // Parse the JSON response from Claude. The SDK decodes each content block2650      // into its concrete class, so find the TextBlock before reading the text.2651      $textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)2652          ?? throw new RuntimeException('Expected a text block in the response.');26532654      return json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);2655  };26562657  // Process the batch of comments and get the response2658  $responseObj = $batchModerateMessages($userComments, $unsafeCategories);26592660  // Print the results for each detected violation2661  foreach ($responseObj['violations'] as $violation) {2662      echo "Comment: {$userComments[$violation['id']]}\n";2663      echo 'Violated Categories: ' . implode(', ', $violation['categories']) . "\n";2664      echo "Explanation: {$violation['explanation']}\n\n";2665  }2666  ```26672668  ```ruby Ruby2669  def batch_moderate_messages(messages, unsafe_categories)2670    # Convert the unsafe categories into a string, with each category on a new line2671    unsafe_category_str = unsafe_categories.join("\n")26722673    # Format the messages string, with each message wrapped in XML-like tags and given an ID2674    messages_str = messages2675      .map.with_index { |message, index| "<message id=#{index}>#{message}</message>" }2676      .join("\n")26772678    # Construct the prompt for Claude, including the messages and unsafe categories2679    assessment_prompt = <<~PROMPT.chomp2680      Determine the messages to moderate, based on the unsafe categories outlined below.26812682      Messages:2683      <messages>2684      #{messages_str}2685      </messages>26862687      Unsafe Categories:2688      <categories>2689      #{unsafe_category_str}2690      </categories>26912692      Respond with ONLY a JSON object, using the format below:2693      {2694        "violations": [2695          {2696            "id": <message id>,2697            "categories": [list of violated categories],2698            "explanation": <Explanation of why there's a violation>2699          }2700        ]2701      }27022703      Important Notes:2704      - Remember to analyze every message for a violation.2705      - Select any number of violations that reasonably apply.2706      - Do not include markdown formatting or code fences in your response.2707    PROMPT27082709    # Send the request to Claude for content moderation2710    response = CLIENT.messages.create(2711      model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs2712      max_tokens: 2048, # Increased max token count to handle batches2713      messages: [{role: :user, content: assessment_prompt}]2714    )27152716    # Parse the JSON response from Claude2717    text_block = response.content.find { it.type == :text }2718    JSON.parse(text_block.text)2719  end272027212722  # Process the batch of comments and get the response2723  response_obj = batch_moderate_messages(USER_COMMENTS, UNSAFE_CATEGORIES)27242725  # Print the results for each detected violation2726  response_obj["violations"].each do |violation|2727    puts <<~RESULT2728      Comment: #{USER_COMMENTS[violation["id"]]}2729      Violated Categories: #{violation["categories"].join(", ")}2730      Explanation: #{violation["explanation"]}27312732    RESULT2733  end2734  ```2735</CodeGroup>27362737In this example, the `batch_moderate_messages` function handles the moderation of an entire batch of messages with a single Claude API call. Inside the function, a prompt is created that includes the list of messages to evaluate and the unsafe content categories. The prompt directs Claude to return a JSON object listing all messages that contain violations. Each message in the response is identified by its `id`, which corresponds to the message's position in the batch. Keep in mind that finding the optimal batch size for your specific needs may require some experimentation. While larger batch sizes can lower costs, they might also lead to a slight decrease in quality. Additionally, you may need to increase the `max_tokens` parameter in the Claude API call to accommodate longer responses. For details on the maximum number of tokens your chosen model can output, refer to the [model comparison table](https://platform.claude.com/docs/en/models/overview#latest-models-comparison).27382739<CardGroup cols={2}>2740  <Card title="Content moderation cookbook" icon="link" href="https://platform.claude.com/cookbook/misc-building-moderation-filter">2741    View a fully implemented code-based example of how to use Claude for content moderation.2742  </Card>27432744  <Card title="Mitigate jailbreaks" icon="link" href="https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks">2745    Explore guardrail techniques to moderate interactions with Claude.2746  </Card>2747</CardGroup>2748
Files main
Menu

Theme