SPB Git

Repository Public

reduce-latency.md 9.5 KB 217 lines markdown

1---2title: Reducing latency3url: https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latency4description: Reduce Claude's response latency by choosing a faster model like Claude Haiku 4.5, trimming prompt and output tokens, and streaming responses.5---67Latency refers to the time it takes for the model to process a prompt and generate an output. Latency can be influenced by various factors, such as the size of the model, the complexity of the prompt, and the underlying infrastructure supporting the model and point of interaction.89<Note>10  It's always better to first engineer a prompt that works well without model or prompt constraints, and then try latency reduction strategies afterward. Trying to reduce latency prematurely might prevent you from discovering what top performance looks like.11</Note>1213***1415## How to measure latency1617When discussing latency, you might come across several terms and measurements:1819* **Baseline latency:** This is the time taken by the model to process the prompt and generate the response, without considering the input and output tokens per second. It provides a general idea of the model's speed.20* **Time to first token (TTFT):** This metric measures the time it takes for the model to generate the first token of the response, from when the prompt was sent. It's particularly relevant when you're using streaming (more on that later) and want to provide a responsive experience to your users.2122For a more in-depth understanding of these terms, check out the [glossary](https://platform.claude.com/docs/en/about-claude/glossary).2324***2526## How to reduce latency2728### 1. Choose the right model2930One of the most direct ways to reduce latency is to select the appropriate model for your use case. Anthropic offers a [range of models](https://platform.claude.com/docs/en/models/overview) with different capabilities and performance characteristics. Consider your specific requirements and choose the model that best fits your needs in terms of speed and output quality.3132For speed-critical applications, **Claude Haiku 4.5** offers the fastest response times while maintaining high intelligence:3334<CodeGroup>35  ```bash cURL36  # For time-sensitive applications, use Claude Haiku 4.537  curl https://api.anthropic.com/v1/messages \38    -H "x-api-key: $ANTHROPIC_API_KEY" \39    -H "anthropic-version: 2023-06-01" \40    -H "content-type: application/json" \41    -d '{42      "model": "claude-haiku-4-5",43      "max_tokens": 100,44      "messages": [{"role": "user", "content": "Summarize this customer feedback in 2 sentences: [feedback text]"}]45    }'46  ```4748  ```bash CLI49  # For time-sensitive applications, use Claude Haiku 4.550  ant messages create \51    --model claude-haiku-4-5 \52    --max-tokens 100 \53    --message '{"role": "user", "content": "Summarize this customer feedback in 2 sentences: [feedback text]"}'54  ```5556  ```python Python57  client = anthropic.Anthropic()5859  # For time-sensitive applications, use Claude Haiku 4.560  message = client.messages.create(61      model="claude-haiku-4-5",62      max_tokens=100,63      messages=[64          {65              "role": "user",66              "content": "Summarize this customer feedback in 2 sentences: [feedback text]",67          }68      ],69  )70  print(message.content[0].text)71  ```7273  ```typescript TypeScript74  const client = new Anthropic();7576  // For time-sensitive applications, use Claude Haiku 4.577  const message = await client.messages.create({78    model: "claude-haiku-4-5",79    max_tokens: 100,80    messages: [81      {82        role: "user",83        content: "Summarize this customer feedback in 2 sentences: [feedback text]"84      }85    ]86  });87  const textBlock = message.content.find((block) => block.type === "text");88  console.log(textBlock?.text);89  ```9091  ```csharp C#92  AnthropicClient client = new();9394  // For time-sensitive applications, use Claude Haiku 4.595  var parameters = new MessageCreateParams96  {97      Model = Model.ClaudeHaiku4_5,98      MaxTokens = 100,99      Messages = [100          new()101          {102              Role = Role.User,103              Content = "Summarize this customer feedback in 2 sentences: [feedback text]"104          }105      ]106  };107  var message = await client.Messages.Create(parameters);108  message.Content[0].TryPickText(out var textBlock);109  Console.WriteLine(textBlock?.Text);110  ```111112  ```go Go113  client := anthropic.NewClient()114115  // For time-sensitive applications, use Claude Haiku 4.5116  message, err := client.Messages.New(context.TODO(), anthropic.MessageNewParams{117  	Model:     anthropic.ModelClaudeHaiku4_5,118  	MaxTokens: 100,119  	Messages: []anthropic.MessageParam{120  		anthropic.NewUserMessage(anthropic.NewTextBlock("Summarize this customer feedback in 2 sentences: [feedback text]")),121  	},122  })123  if err != nil {124  	log.Fatal(err)125  }126  fmt.Println(message.Content[0].Text)127  ```128129  ```java Java130  AnthropicClient client = AnthropicOkHttpClient.fromEnv();131132  // For time-sensitive applications, use Claude Haiku 4.5133  MessageCreateParams params = MessageCreateParams.builder()134      .model(Model.CLAUDE_HAIKU_4_5)135      .maxTokens(100L)136      .addUserMessage("Summarize this customer feedback in 2 sentences: [feedback text]")137      .build();138  Message message = client.messages().create(params);139  IO.println(message.content().get(0).text().map(TextBlock::text).orElse(""));140  ```141142  ```php PHP143  $client = new Client();144145  // For time-sensitive applications, use Claude Haiku 4.5146  $message = $client->messages->create(147      maxTokens: 100,148      messages: [['role' => 'user', 'content' => 'Summarize this customer feedback in 2 sentences: [feedback text]']],149      model: 'claude-haiku-4-5',150  );151  echo $message->content[0]->text;152  ```153154  ```ruby Ruby155  client = Anthropic::Client.new156157  # For time-sensitive applications, use Claude Haiku 4.5158  message = client.messages.create(159    model: "claude-haiku-4-5",160    max_tokens: 100,161    messages: [{ role: "user", content: "Summarize this customer feedback in 2 sentences: [feedback text]" }]162  )163  puts message.content.first.text164  ```165</CodeGroup>166167For more details about model metrics, see the [models overview](https://platform.claude.com/docs/en/models/overview) page.168169### 2. Optimize prompt and output length170171Minimize the number of tokens in both your input prompt and the expected output, while still maintaining high performance. The fewer tokens the model has to process and generate, the faster the response will be.172173Here are some tips to help you optimize your prompts and outputs:174175* **Be clear but concise:** Aim to convey your intent clearly and concisely in the prompt. Avoid unnecessary details or redundant information, while keeping in mind that [Claude lacks context](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#be-clear-and-direct) on your use case and might not make the intended leaps of logic if instructions are unclear.176* **Ask for shorter responses:** Ask Claude directly to be concise. If Claude is outputting unwanted length, ask Claude to [curb its chattiness](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#be-clear-and-direct).177  <Tip>178    Because of how LLMs count 179180    [tokens](https://platform.claude.com/docs/en/about-claude/glossary#tokens)181182     instead of words, asking for an exact word count or a word count limit is not as effective a strategy as asking for paragraph or sentence count limits.183  </Tip>184* **Set appropriate output limits:** Use the `max_tokens` parameter to set a hard limit on the maximum length of the generated response. This prevents Claude from generating overly long outputs.185  <Note>186    When the response reaches 187188    `max_tokens`189190     tokens, the response will be cut off, perhaps mid-sentence or mid-word, so this is a blunt technique that might require post-processing and is usually most appropriate for multiple choice or short answer responses where the answer comes right at the beginning.191  </Note>192* **Experiment with temperature:** The `temperature` [parameter](https://platform.claude.com/docs/en/api/messages/create) controls the randomness of the output. Lower values (for example, 0.2) can sometimes lead to more focused and shorter responses, while higher values (for example, 0.8) might result in more diverse but potentially longer outputs.193194Finding the right balance among prompt clarity, output quality, and token count might require some experimentation.195196### 3. Stream responses197198Streaming is a feature that allows the model to start sending back its response before the full output is complete. This can significantly improve the perceived responsiveness of your application, as users can see the model's output in real time.199200With streaming enabled, you can process the model's output as it arrives, updating your user interface or performing other tasks in parallel.201202Visit [Streaming messages](https://platform.claude.com/docs/en/build-with-claude/streaming) to learn about how you can implement streaming for your use case.203204***205206## Next steps207208<CardGroup cols={2}>209  <Card title="Reduce hallucinations" icon="shield" href="https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations">210    Minimize hallucinations in Claude's outputs by allowing uncertainty, grounding responses in direct quotes, and verifying claims with citations.211  </Card>212213  <Card title="Streaming messages" icon="bolt" href="https://platform.claude.com/docs/en/build-with-claude/streaming">214    Stream Messages API responses incrementally with server-sent events, including text, tool use, and extended thinking deltas.215  </Card>216</CardGroup>217
Files main
Menu

Theme