Repository Public
1# Latency optimization23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45This guide covers the core set of principles you can apply to improve latency across a wide variety of LLM-related use cases. These techniques come from working with a wide range of customers and developers on production applications, so they should apply regardless of what you're building—from a granular workflow to an end-to-end chat application.67Although there are many individual techniques, this guide groups them into **seven principles** that represent a high-level taxonomy of approaches for improving latency.89At the end, we'll walk through an [example](#example) to see how they can be applied.1011### Seven principles12131. [Process tokens faster.](#process-tokens-faster)142. [Generate fewer tokens.](#generate-fewer-tokens)153. [Use fewer input tokens.](#use-fewer-input-tokens)164. [Make fewer requests.](#make-fewer-requests)175. [Parallelize.](#parallelize)186. [Make your users wait less.](#make-your-users-wait-less)197. [Don't default to an LLM.](#dont-default-to-an-llm)2021## Process tokens faster2223**Inference speed** is probably the first thing that comes to mind when addressing latency (but as you'll see soon, it's far from the only one). This refers to the actual **rate at which the LLM processes tokens**, and is often measured in TPM (tokens per minute) or TPS (tokens per second).2425The main factor that influences inference speed is **model size**—smaller models usually run faster (and cheaper), and when used correctly can even outperform larger models. To maintain high-quality performance with smaller models, you can explore:2627- using a longer, [more detailed prompt](https://developers.openai.com/api/docs/guides/prompt-engineering#prompt-engineering),28- adding (more) [few-shot examples](https://developers.openai.com/api/docs/guides/prompt-engineering#few-shot-learning), or29- [fine-tuning](https://developers.openai.com/api/docs/guides/model-optimization) / distillation.3031You can also employ inference optimizations like our [**Predicted outputs**](https://developers.openai.com/api/docs/guides/predicted-outputs) feature. Predicted outputs let you significantly reduce latency of a generation when you know most of the output ahead of time, such as code editing tasks. By giving the model a prediction, the LLM can focus more on the actual changes, and less on the content that will remain the same.32333435Other factors that affect inference speed are the amount of 36 **compute** you have available and any additional 37 **inference optimizations** you employ. 38 3940 Most people can't influence these factors directly, but if you're curious, and41 have some control over your infra, **faster hardware** or 42 **running engines at a lower saturation** may give you a modest43 TPM boost. And if you're down in the trenches, there's a myriad of other 44 [inference optimizations](https://lilianweng.github.io/posts/2023-01-10-inference-optimization/) 45 that are a bit beyond the scope of this guide.46474849## Generate fewer tokens5051Generating tokens is almost always the highest latency step when using an LLM: as a general heuristic, **cutting 50% of your output tokens may cut ~50% of your latency**. The way you reduce your output size will depend on output type:5253If you're generating **natural language**, **asking the model to be more concise** ("under 20 words" or "be brief") may help. You can also use few-shot examples and/or fine-tuning to teach the model shorter responses.5455If you're generating **structured output**, try to **minimize your output syntax** where possible: shorten function names, omit named arguments, coalesce parameters, etc.5657Finally, while not common, you can also use `max_tokens` or `stop_tokens` to end your generation early.5859Always remember: an output token cut is a (milli)second earned!6061## Use fewer input tokens6263While reducing the number of input tokens does result in lower latency, this is not usually a significant factor—**cutting 50% of your prompt may only result in a 1–5% latency improvement**. Unless you're working with truly massive context sizes (documents, images), you may want to spend your efforts elsewhere.6465That being said, if you _are_ working with massive contexts (or you're set on squeezing every last bit of performance _and_ you've exhausted all other options) you can use the following techniques to reduce your input tokens:6667- **Fine-tuning the model**, to replace the need for lengthy instructions / examples.68- **Filtering context input**, like pruning RAG results, cleaning HTML, etc.69- **Maximize shared prompt prefix**, by putting dynamic portions (for example, RAG results and history) later in the prompt. This makes your request more [KV cache](https://medium.com/@joaolages/kv-caching-explained-276520203249)-friendly (which most LLM providers use) and means fewer input tokens are processed on each request.7071Check out our docs to learn more about how [prompt72 caching](https://developers.openai.com/api/docs/guides/prompt-engineering#save-on-cost-and-latency-with-prompt-caching)73 works.7475## Make fewer requests7677Each time you make a request, you incur some round-trip latency—this can start to add up.7879If you have sequential steps for the LLM to perform, instead of firing off one request per step consider **putting them in a single prompt and getting them all in a single response**. You'll avoid the additional round-trip latency, and potentially also reduce complexity of processing multiple responses.8081An approach to doing this is by collecting your steps in an enumerated list in the combined prompt, and then requesting the model to return the results in named fields in a JSON object. This way, you can parse and reference each result.8283## Parallelize8485Parallel processing can be powerful when performing multiple steps with an LLM.8687If the steps **are _not_ strictly sequential**, you can **split them out into parallel calls**. Two shirts take just as long to dry as one.8889If the steps **_are_ strictly sequential**, however, you might still be able to **leverage speculative execution**. This is particularly effective for classification steps where one outcome is more likely than the others (for example, moderation).90911. Start step 1 and step 2 simultaneously (for example, input moderation and story generation)922. Verify the result of step 1933. If result was not the expected, cancel step 2 (and retry if necessary)9495If your guess for step 1 is right, then you essentially got to run it with zero added latency!9697## Make your users wait less9899There's a huge difference between **waiting** and **watching progress happen**—make sure your users experience the latter. Here are a few techniques:100101- **Streaming**: The single most effective approach, as it cuts the _waiting_ time to a second or less. (ChatGPT would feel pretty different if you saw nothing until each response was done.)102- **Chunking**: If your output needs further processing before being shown to the user (moderation, translation), consider **processing it in chunks** instead of all at once. Do this by streaming to your back end, then sending processed chunks to your front end.103- **Show your steps**: If you're taking multiple steps or using tools, surface this to the user. The more real progress you can show, the better.104- **Loading states**: Spinners and progress bars go a long way.105106Note that while **showing your steps & having loading states** have a mostly107psychological effect, **streaming & chunking** genuinely do reduce overall108latency once you consider the app + user system: the user will finish reading a response109sooner.110111## Don't default to an LLM112113Language models are powerful and versatile, and are therefore sometimes used in cases where a **faster classical method** would be more appropriate. Identifying such cases may allow you to cut your latency significantly. Consider the following examples:114115- **Hard-coding:** If your **output** is highly constrained, you may not need an LLM to generate it. Action confirmations, refusal messages, and requests for standard input are all great candidates to be hard-coded. (You can even use the age-old method of coming up with a few variations for each.)116- **Pre-computing:** If your **input** is constrained (for example, category selection), you can generate multiple responses in advance, and just make sure you never show the same one to a user twice.117- **Leveraging UI:** Summarized metrics, reports, or search results are sometimes better conveyed with classical, bespoke UI components rather than LLM-generated text.118- **Traditional optimization techniques:** An LLM application is still an application; binary search, caching, hash maps, and runtime complexity are all _still_ useful in a world of language models.119120## Example121122Let's now look at a sample application, identify potential latency optimizations, and propose some solutions!123124We'll be analyzing the architecture and prompts of a hypothetical customer service bot inspired by real production applications. The [architecture and prompts](#architecture-and-prompts) section sets the stage, and the [analysis and optimizations](#analysis-and-optimizations) section will walk through the latency optimization process.125126You'll notice this example doesn't cover every single principle, much like127 real-world use cases don't require applying every technique.128129### Architecture and prompts130131The following is the **initial architecture** for a hypothetical **customer service bot**. This is what we'll be making changes to.132133134135At a high level, the diagram flow describes the following process:1361371. A user sends a message as part of an ongoing conversation.1382. The last message is turned into a **self-contained query** (see examples in prompt).1393. We determine whether or not **additional (retrieved) information is required** to respond to that query.1404. **Retrieval** is performed, producing search results.1415. The assistant **reasons** about the user's query and search results, and **produces a response**.1426. The response is sent back to the user.143144Below are the prompts used in each part of the diagram. While they are still only hypothetical and simplified, they are written with the same structure and wording that you would find in a production application.145146Places where you see placeholders like "**[user input here]**" represent147 dynamic portions, that would be replaced by actual data at runtime.148149150151#### Query contextualization prompt152153154155Re-writes user query to be a self-contained search query.156157```example-chat158SYSTEM: Given the previous conversation, re-write the last user query so it contains159all necessary context.160161# Example162History: [{user: "What is your return policy?"},{assistant: "..."}]163User Query: "How long does it cover?"164Response: "How long does the return policy cover?"165166# Conversation167[last 3 messages of conversation]168169# User Query170[last user query]171172USER: [JSON-formatted input conversation here]173```174175176177178179180181#### Retrieval check prompt182183184Determines whether a query requires performing retrieval to respond.185186```example-chat187SYSTEM: Given a user query, determine whether it requires doing a realtime lookup to188respond to.189190# Examples191User Query: "How can I return this item after 30 days?"192Response: "true"193194User Query: "Thank you!"195Response: "false"196197USER: [input user query here]198```199200201202203204205206#### Assistant prompt207208209Fills the fields of a JSON to reason through a pre-defined set of steps to produce a final response given a user conversation and relevant retrieved information.210211```example-chat212SYSTEM: You are a helpful customer service bot.213214Use the result JSON to reason about each user query - use the retrieved context.215216# Example217218User: "My computer screen is cracked! I want it fixed now!!!"219220Assistant Response:221{222 "message_is_conversation_continuation": "True",223 "number_of_messages_in_conversation_so_far": "1",224 "user_sentiment": "Aggravated",225 "query_type": "Hardware Issue",226 "response_tone": "Validating and solution-oriented",227 "response_requirements": "Propose options for repair or replacement.",228 "user_requesting_to_talk_to_human": "False",229 "enough_information_in_context": "True",230 "response": "..."231}232233USER: # Relevant Information234` ` `235[retrieved context]236` ` `237238USER: [input user query here]239```240241242243244245### Analysis and optimizations246247#### Part 1: Looking at retrieval prompts248249Looking at the architecture, the first thing that stands out is the **consecutive GPT-4 calls** - these hint at a potential inefficiency, and can often be replaced by a single call or parallel calls.250251252253In this case, since the check for retrieval requires the contextualized query, let's **combine them into a single prompt** to [make fewer requests](#make-fewer-requests).254255256257258259##### Combined query contextualization and retrieval check prompt260261262263**What changed?** Before, we had one prompt to re-write the query and one to determine whether this requires doing a retrieval lookup. Now, this combined prompt does both. Specifically, notice the updated instruction in the first line of the prompt, and the updated output JSON:264265```javascript266{267 query: "[contextualized query]",268 retrieval: "[true/false - whether retrieval is required]",269}270```271272```ruby273combined_query = {274 query: "[contextualized query]",275 retrieval: "[true/false - whether retrieval is required]"276}277278puts(combined_query)279```280281282```example-chat283SYSTEM: Given the previous conversation, re-write the last user query so it contains284all necessary context. Then, determine whether the full request requires doing a285realtime lookup to respond to.286287Respond in the following form:288{289 query:"[contextualized query]",290 retrieval:"[true/false - whether retrieval is required]"291}292293# Examples294295History: [{user: "What is your return policy?"},{assistant: "..."}]296User Query: "How long does it cover?"297Response: {query: "How long does the return policy cover?", retrieval: "true"}298299History: [{user: "How can I return this item after 30 days?"},{assistant: "..."}]300User Query: "Thank you!"301Response: {query: "Thank you!", retrieval: "false"}302303# Conversation304[last 3 messages of conversation]305306# User Query307[last user query]308309USER: [JSON-formatted input conversation here]310```311312313314315316317318Actually, adding context and determining whether to retrieve are straightforward and well-defined tasks, so we can likely use a **smaller, fine-tuned model** instead. Switching to GPT-3.5 will let us [process tokens faster](#process-tokens-faster).319320321322#### Part 2: Analyzing the assistant prompt323324Let's now direct our attention to the Assistant prompt. There seem to be many distinct steps happening as it fills the JSON fields—this could indicate an opportunity to [parallelize](#parallelize).325326327328However, let's pretend we have run some tests and discovered that splitting the reasoning steps in the JSON produces worse responses, so we need to explore different solutions.329330**Could we use a fine-tuned GPT-3.5 instead of GPT-4?** Maybe—but in general, open-ended responses from assistants are best left to GPT-4 so it can better handle a greater range of cases. That being said, looking at the reasoning steps themselves, they may not all require GPT-4-level reasoning to produce. Their well-defined, limited scope makes them **good potential candidates for fine-tuning**.331332```javascript333{334 message_is_conversation_continuation: "True", // <-335 number_of_messages_in_conversation_so_far: "1", // <-336 user_sentiment: "Aggravated", // <-337 query_type: "Hardware Issue", // <-338 response_tone: "Validating and solution-oriented", // <-339 response_requirements: "Propose options for repair or replacement.", // <-340 user_requesting_to_talk_to_human: "False", // <-341 enough_information_in_context: "True", // <-342 response: "...", // X -- benefits from GPT-4343}344```345346```ruby347assistant_response = {348 message_is_conversation_continuation: "True", # <-349 number_of_messages_in_conversation_so_far: "1", # <-350 user_sentiment: "Aggravated", # <-351 query_type: "Hardware Issue", # <-352 response_tone: "Validating and solution-oriented", # <-353 response_requirements: "Propose options for repair or replacement.", # <-354 user_requesting_to_talk_to_human: "False", # <-355 enough_information_in_context: "True", # <-356 response: "..." # X -- benefits from GPT-4357}358359puts(assistant_response)360```361362363This opens up the possibility of a trade-off. Do we keep this as a **single request entirely generated by GPT-4**, or **split it into two sequential requests** and use GPT-3.5 for all but the final response? We have a case of conflicting principles: the first option lets us [make fewer requests](#make-fewer-requests), but the second may let us [process tokens faster](#process-tokens-faster).364365As with many optimization tradeoffs, the answer will depend on the details. For example:366367- The proportion of tokens in the `response` vs the other fields.368- The average latency decrease from processing most fields faster.369- The average latency _increase_ from doing two requests instead of one.370371The conclusion will vary by case, and the best way to make the determination is by testing this with production examples. In this case, let's pretend the tests indicated it's favorable to split the prompt in two to [process tokens faster](#process-tokens-faster).372373374375**Note:** We'll be grouping `response` and `enough_information_in_context` together in the second prompt to avoid passing the retrieved context to both new prompts.376377378379##### Assistants prompt - reasoning380381382383This prompt will be passed to GPT-3.5 and can be fine-tuned on curated examples.384385**What changed?** The "enough_information_in_context" and "response" fields were removed, and the retrieval results are no longer loaded into this prompt.386387```example-chat388SYSTEM: You are a helpful customer service bot.389390Based on the previous conversation, respond in a JSON to determine the required391fields.392393# Example394395User: "My freaking computer screen is cracked!"396397Assistant Response:398{399 "message_is_conversation_continuation": "True",400 "number_of_messages_in_conversation_so_far": "1",401 "user_sentiment": "Aggravated",402 "query_type": "Hardware Issue",403 "response_tone": "Validating and solution-oriented",404 "response_requirements": "Propose options for repair or replacement.",405 "user_requesting_to_talk_to_human": "False",406}407```408409410411412413414##### Assistants prompt - response415416417418This prompt will be processed by GPT-4 and will receive the reasoning steps determined in the prior prompt, as well as the results from retrieval.419420**What changed?** All steps were removed except for "enough_information_in_context" and "response". Additionally, the JSON we were previously filling in as output will be passed in to this prompt.421422```example-chat423SYSTEM: You are a helpful customer service bot.424425Use the retrieved context, as well as these pre-classified fields, to respond to426the user's query.427428# Reasoning Fields429` ` `430[reasoning json determined in previous GPT-3.5 call]431` ` `432433# Example434435User: "My freaking computer screen is cracked!"436437Assistant Response:438{439 "enough_information_in_context": "True",440 "response": "..."441}442443USER: # Relevant Information444` ` `445[retrieved context]446` ` `447```448449450451452453454455456In fact, now that the reasoning prompt does not depend on the retrieved context we can [parallelize](#parallelize) and fire it off at the same time as the retrieval prompts.457458459460#### Part 3: Optimizing the structured output461462Let's take another look at the reasoning prompt.463464465466Taking a closer look at the reasoning JSON you may notice the field names themselves are quite long.467468```javascript469{470 message_is_conversation_continuation: "True", // <-471 number_of_messages_in_conversation_so_far: "1", // <-472 user_sentiment: "Aggravated", // <-473 query_type: "Hardware Issue", // <-474 response_tone: "Validating and solution-oriented", // <-475 response_requirements: "Propose options for repair or replacement.", // <-476 user_requesting_to_talk_to_human: "False", // <-477}478```479480```ruby481reasoning = {482 message_is_conversation_continuation: "True", # <-483 number_of_messages_in_conversation_so_far: "1", # <-484 user_sentiment: "Aggravated", # <-485 query_type: "Hardware Issue", # <-486 response_tone: "Validating and solution-oriented", # <-487 response_requirements: "Propose options for repair or replacement.", # <-488 user_requesting_to_talk_to_human: "False" # <-489}490491puts(reasoning)492```493494495By making them shorter and moving explanations to the comments we can [generate fewer tokens](#generate-fewer-tokens).496497```javascript498{499 cont: "True", // whether last message is a continuation500 n_msg: "1", // number of messages in the continued conversation501 tone_in: "Aggravated", // sentiment of user query502 type: "Hardware Issue", // type of the user query503 tone_out: "Validating and solution-oriented", // desired tone for response504 reqs: "Propose options for repair or replacement.", // response requirements505 human: "False", // whether user is expressing want to talk to human506}507```508509```ruby510reasoning = {511 cont: "True", # whether last message is a continuation512 n_msg: "1", # number of messages in the continued conversation513 tone_in: "Aggravated", # sentiment of user query514 type: "Hardware Issue", # type of the user query515 tone_out: "Validating and solution-oriented", # desired tone for response516 reqs: "Propose options for repair or replacement.", # response requirements517 human: "False" # whether user wants to talk to a human518}519520puts(reasoning)521```522523524525526This small change removed 19 output tokens. While with GPT-3.5 this may only result in a few millisecond improvement, with GPT-4 this could shave off up to a second.527528529530You might imagine, however, how this can have quite a significant impact for larger model outputs.531532We could go further and use single characters for the JSON fields, or put everything in an array, but this may start to hurt our response quality. The best way to know, once again, is through testing.533534#### Example wrap-up535536Let's review the optimizations we implemented for the customer service bot example:5375385395401. **Combined** query contextualization and retrieval check steps to [make fewer requests](#make-fewer-requests).5412. For the new prompt, **switched to a smaller, fine-tuned GPT-3.5** to [process tokens faster](#process-tokens-faster).5423. Split the assistant prompt in two, **switching to a smaller, fine-tuned GPT-3.5** for the reasoning, again to [process tokens faster](#process-tokens-faster).5434. [Parallelized](#parallelize) the retrieval checks and the reasoning steps.5445. **Shortened reasoning field names** and moved comments into the prompt, to [generate fewer tokens](#generate-fewer-tokens).