spb/doc-api

1# Long context23Many Gemini models come with large context windows of 1 million or more tokens.4Historically, large language models (LLMs) were significantly limited by5the amount of text (or tokens) that could be passed to the model at one time.6The Gemini long context window unlocks many new use cases and developer7paradigms.89The code you already use for cases like [text10generation](https://ai.google.dev/gemini-api/docs/text-generation) or [multimodal11inputs](https://ai.google.dev/gemini-api/docs/vision) will work without any changes with long context.1213This document gives you an overview of what you can achieve using models with14context windows of 1M and more tokens. The page gives a brief overview of15a context window, and explores how developers should think about long context,16various real world use cases for long context, and ways to optimize the usage17of long context.1819For the context window sizes of specific models, see the20[Models](https://ai.google.dev/gemini-api/docs/models) page.2122## What is a context window?2324The basic way you use the Gemini models is by passing information (context)25to the model, which will subsequently generate a response. An analogy for the26context window is short term memory. There is a limited amount of information27that can be stored in someone's short term memory, and the same is true for28generative models.2930You can read more about how models work under the hood in our [generative models31guide](https://ai.google.dev/gemini-api/docs/prompting-strategies#under-the-hood).3233## Getting started with long context3435Earlier versions of generative models were only able to process 8,00036tokens at a time. Newer models pushed this further by accepting 32,000 or even37128,000 tokens. Gemini is the first model capable of accepting 1 million tokens.3839In practice, 1 million tokens would look like:4041- 50,000 lines of code (with the standard 80 characters per line)42- All the text messages you have sent in the last 5 years43- 8 average length English novels44- Transcripts of over 200 average length podcast episodes4546The more limited context windows common in many other models often require47strategies like arbitrarily dropping old messages, summarizing content, using48RAG with vector databases, or filtering prompts to save tokens.4950While these techniques remain valuable in specific scenarios, Gemini's extensive51context window invites a more direct approach: providing all relevant52information upfront. Because Gemini models were purpose-built with massive53context capabilities, they demonstrate powerful in-context learning. For54example, using only in-context instructional materials (a 500-page reference55grammar, a dictionary, and ≈400 parallel sentences), Gemini56[learned to translate](https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf)57from English to Kalamang---a Papuan language with58fewer than 200 speakers---with quality similar to a human learner using the same59materials. This illustrates the paradigm shift enabled by Gemini's long context,60empowering new possibilities through robust in-context learning.6162## Long context use cases6364While the standard use case for most generative models is still text input, the65Gemini model family enables a new paradigm of multimodal use cases. These66models can natively understand text, video, audio, and images. They are67accompanied by the [Gemini API that takes in multimodal file68types](https://ai.google.dev/gemini-api/docs/prompting_with_media) for69convenience.7071### Long form text7273Text has proved to be the layer of intelligence underpinning much of the74momentum around LLMs. As mentioned earlier, much of the practical limitation of75LLMs was because of not having a large enough context window to do certain76tasks. This led to the rapid adoption of retrieval augmented generation (RAG)77and other techniques which dynamically provide the model with relevant78contextual information. Now, with larger and larger context windows, there are79new techniques becoming available which unlock new use cases.8081Some emerging and standard use cases for text based long context include:8283- Summarizing large corpuses of text84  - Previous summarization options with smaller context models would require a sliding window or another technique to keep state of previous sections as new tokens are passed to the model85- Question and answering86  - Historically this was only possible with RAG given the limited amount of context and models' factual recall being low87- Agentic workflows88  - Text is the underpinning of how agents keep state of what they have done and what they need to do; not having enough information about the world and the agent's goal is a limitation on the reliability of agents8990[Many-shot in-context learning](https://arxiv.org/pdf/2404.11018) is one of the91most unique capabilities unlocked by long context models. Research has shown92that taking the common "single shot" or "multi-shot" example paradigm, where the93model is presented with one or a few examples of a task, and scaling that up to94hundreds, thousands, or even hundreds of thousands of examples, can lead to95novel model capabilities. This many-shot approach has also been shown to perform96similarly to models which were fine-tuned for a specific task. For use cases97where a Gemini model's performance is not yet sufficient for a production98rollout, you can try the many-shot approach. As you might explore later in the99long context optimization section, context caching makes this type of high input100token workload much more economically feasible and even lower latency in some101cases.102103### Long form video104105Video content's utility has long been constrained by the lack of accessibility106of the medium itself. It was hard to skim the content, transcripts often failed107to capture the nuance of a video, and most tools don't process image, text, and108audio together. With Gemini, the long-context text capabilities translate to109the ability to reason and answer questions about multimodal inputs with110sustained performance.111112Some emerging and standard use cases for video long context include:113114- Video question and answering115- Video memory, as shown with [Google's Project Astra](https://deepmind.google/technologies/gemini/project-astra/)116- Video captioning117- Video recommendation systems, by enriching existing metadata with new multimodal understanding118- Video customization, by looking at a corpus of data and associated video metadata and then removing parts of videos that are not relevant to the viewer119- Video content moderation120- Real-time video processing121122When working with videos, it is important to consider how the [videos are123processed into tokens](https://ai.google.dev/gemini-api/docs/tokens#media-token), which affects124billing and usage limits. You can learn more about prompting with video files in125the [Prompting126guide](https://ai.google.dev/gemini-api/docs/prompting_with_media?lang=python#prompting-with-videos).127128### Long form audio129130The Gemini models were the first natively multimodal large language models131that could understand audio. Historically, the typical developer workflow would132involve stringing together multiple domain specific models, like a133speech-to-text model and a text-to-text model, in order to process audio. This134led to additional latency required by performing multiple round-trip requests135and decreased performance usually attributed to disconnected architectures of136the multiple model setup.137138Some emerging and standard use cases for audio context include:139140- Real-time transcription and translation141- Podcast / video question and answering142- Meeting transcription and summarization143- Voice assistants144145You can learn more about prompting with audio files in the [Prompting146guide](https://ai.google.dev/gemini-api/docs/prompting_with_media?lang=python#prompting-with-videos).147148## Long context optimizations149150The primary optimization when working with long context and the Gemini151models is to use [context152caching](https://ai.google.dev/gemini-api/docs/caching). Beyond the previous153impossibility of processing lots of tokens in a single request, the other main154constraint was the cost. If you have a "chat with your data" app where a user155uploads 10 PDFs, a video, and some work documents, you would historically have156to work with a more complex retrieval augmented generation (RAG) tool /157framework in order to process these requests and pay a significant amount for158tokens moved into the context window. Now, you can cache the files the user159uploads and pay to store them on a per hour basis. The input / output cost per160request with Gemini Flash for example is \~4x less than the standard161input / output cost, so if162the user chats with their data enough, it becomes a huge cost saving for you as163the developer.164165## Long context limitations166167In various sections of this guide, we talked about how Gemini models achieve168high performance across various needle-in-a-haystack retrieval evals. These169tests consider the most basic setup, where you have a single needle you are170looking for. In cases where you might have multiple "needles" or specific pieces171of information you are looking for, the model does not perform with the same172accuracy. Performance can vary to a wide degree depending on the context. This173is important to consider as there is an inherent tradeoff between getting the174right information retrieved and cost. You can get \~99% on a single query, but175you have to pay the input token cost every time you send that query. So for 100176pieces of information to be retrieved, if you needed 99% performance, you would177likely need to send 100 requests. This is a good example of where context178caching can significantly reduce the cost associated with using Gemini models179while keeping the performance high.180181## FAQs182183### Where is the best place to put my query in the context window?184185In most cases, especially if the total context is long, the model's186performance will be better if you put your query / question at the end of the187prompt (after all the other context).188189### Do I lose model performance when I add more tokens to a query?190191Generally, if you don't need tokens to be passed to the model, it is best to192avoid passing them. However, if you have a large chunk of tokens with some193information and want to ask questions about that information, the model is194highly capable of extracting that information (up to 99% accuracy in many195cases).196197### How can I lower my cost with long-context queries?198199If you have a similar set of tokens / context that you want to re-use many200times, [context caching](https://ai.google.dev/gemini-api/docs/caching) can help reduce the costs201associated with asking questions about that information.202203### Does the context length affect the model latency?204205There is some fixed amount of latency in any given request, regardless of the206size, but generally longer queries will have higher latency (time to first207token).