Repository Public
1Description: Learn how to optimize costs with the Flex inference tier23The Gemini Flex API is an inference tier that offers a 50% cost reduction4compared to standard rates, in exchange for variable latency and best-effort5availability. It's designed for latency-tolerant workloads that require6synchronous processing but don't need the real-time performance of the standard7API.89## How to use Flex1011To use the Flex tier, specify the `service_tier` as `flex` in the12request body. By default, requests use the standard tier if this field is13omitted.1415### Python1617 from google import genai1819 client = genai.Client()2021 try:22 response = client.models.generate_content(23 model="gemini-3.8-flash",24 contents="Analyze this dataset for trends...",25 config={"service_tier": "flex"},26 )27 print(response.text)28 except Exception as e:29 print(f"Flex request failed: {e}")3031### JavaScript3233 import {GoogleGenAI} from '@google/genai';3435 const ai = new GoogleGenAI({});3637 async function main() {38 try {39 const response = await ai.models.generateContent({40 model: "gemini-3.8-flash",41 contents: "Analyze this dataset for trends...",42 config: { serviceTier: "flex" },43 });44 console.log(response.text);45 } catch (e) {46 console.log(`Flex request failed: ${e}`);47 }48 }4950 await main();5152### Go5354 package main5556 import (57 "context"58 "fmt"59 "log"60 "google.golang.org/genai"61 )6263 func main() {64 ctx := context.Background()65 client, err := genai.NewClient(ctx, nil)66 if err != nil {67 log.Fatal(err)68 }6970 result, err := client.Models.GenerateContent(71 ctx,72 "gemini-3.8-flash",73 genai.Text("Analyze this dataset for trends..."),74 &genai.GenerateContentConfig{75 ServiceTier: "flex",76 },77 )78 if err != nil {79 log.Printf("Flex request failed: %v", err)80 return81 }82 fmt.Println(result.Text())83 }8485### REST8687 curl -X POST "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.8-flash:generateContent?key=$GEMINI_API_KEY" \88 -H "Content-Type: application/json" \89 -d '{90 "contents": [{91 "parts":[{"text": "Summarize the latest research on quantum computing."}]92 }],93 "service_tier": "flex"94 }'9596## How Flex inference works9798Gemini Flex inference bridges the gap between the standard API and the 24-hour99turnaround of the [Batch API](https://ai.google.dev/gemini-api/docs/batch-api). It utilizes off-peak,100"sheddable" compute capacity to provide a cost-effective solution for background101tasks and sequential workflows.102103| Feature | Flex | Priority | Standard | Batch |104|---|---|---|---|---|105| **Pricing** | 50% discount | 75-100% more than Standard | Full price | 50% discount |106| **Latency** | Minutes (1--15 min target) | Low (Seconds) | Seconds to minutes | Up to 24 hours |107| **Reliability** | Best-effort (Sheddable) | High (Non-sheddable) | High / Medium-high | High (for throughput) |108| **Interface** | Synchronous | Synchronous | Synchronous | Asynchronous |109110### Key benefits111112- **Cost efficiency**: Substantial savings for non-production evals, background agents, and data enrichment.113- **Low friction**: No need to manage batch objects, job IDs, or polling; simply add a single parameter to your existing requests.114- **Synchronous workflows**: Ideal for sequential API chains where the next request depends on the output of the previous one, making it more flexible than Batch for agentic workflows.115116### Use cases117118- **Offline evaluations**: Running "LLM-as-a-judge" regression tests or leaderboards.119- **Background agents**: Sequential tasks like CRM updates, profile building, or content moderation where minutes of delay are acceptable.120- **Budget-constrained research**: Academic experiments that require high token volume on a limited budget.121122### Rate limits123124Flex inference traffic counts towards your general [rate limits](https://aistudio.google.com/rate-limit); it doesn't125offer extended rate limits like the [Batch API](https://ai.google.dev/gemini-api/docs/batch-api).126127### Sheddable capacity128129Flex traffic is treated with lower priority. If there is a spike in130standard traffic, Flex requests may be preempted or evicted to ensure capacity131for high-priority users. If you're looking for high-priority inference, check132[Priority inference](https://ai.google.dev/gemini-api/docs/priority-inference)133134### Error codes135136When Flex capacity is unavailable or the system is congested, the API will137return standard error codes:138139- **503 Service Unavailable**: The system is currently at capacity.140- **429 Too Many Requests**: Rate limits or resource exhaustion.141142### Client responsibility143144- **No server-side fallback**: To prevent unexpected charges, the system won't automatically upgrade a Flex request to the Standard tier if Flex capacity is full.145- **Retries**: You must implement your own client-side retry logic with exponential backoff.146- **Timeouts**: Because Flex requests may sit in a queue, we recommend increasing client-side timeouts to 10 minutes or more to avoid premature connection closure.147148## Adjust timeout windows149150You can configure per-request timeouts for the REST API and client libraries,151and global timeouts only when using the client libraries.152153Always ensure your client-side timeout covers the intended server patience154window (e.g., 600s+ for Flex wait queues). The SDKs expect timeout values in155milliseconds.156157### Per-request timeouts158159### Python160161 from google import genai162163 client = genai.Client()164165 try:166 response = client.models.generate_content(167 model="gemini-3.8-flash",168 contents="why is the sky blue?",169 config={170 "service_tier": "flex",171 "http_options": {"timeout": 900000}172 },173 )174 except Exception as e:175 print(f"Flex request failed: {e}")176177 # Example with streaming178 try:179 response = client.models.generate_content_stream(180 model="gemini-3.8-flash",181 contents=["List 5 ideas for a sci-fi movie."],182 config={183 "service_tier": "flex",184 "http_options": {"timeout": 60000}185 }186 # Per-request timeout for the streaming operation187 )188 for chunk in response:189 print(chunk.text, end="")190191 except Exception as e:192 print(f"An error occurred during streaming: {e}")193194### JavaScript195196 import {GoogleGenAI} from '@google/genai';197198 const client = new GoogleGenAI({});199200 async function main() {201 try {202 const response = await client.models.generateContent({203 model: "gemini-3.8-flash",204 contents: "why is the sky blue?",205 config: {206 serviceTier: "flex",207 httpOptions: {timeout: 900000}208 },209 });210 } catch (e) {211 console.log(`Flex request failed: ${e}`);212 }213214 // Example with streaming215 try {216 const response = await client.models.generateContentStream({217 model: "gemini-3.8-flash",218 contents: ["List 5 ideas for a sci-fi movie."],219 config: {220 serviceTier: "flex",221 httpOptions: {timeout: 60000}222 },223 });224 for await (const chunk of response.stream) {225 process.stdout.write(chunk.text());226 }227 } catch (e) {228 console.log(`An error occurred during streaming: ${e}`);229 }230 }231232 await main();233234### Go235236 package main237238 import (239 "context"240 "fmt"241 "log"242 "time"243244 "google.golang.org/api/iterator"245 "google.golang.org/genai"246 )247248 func main() {249 ctx := context.Background()250 client, err := genai.NewClient(ctx, nil)251 if err != nil {252 log.Fatal(err)253 }254 defer client.Close()255256 timeoutCtx, cancel := context.WithTimeout(ctx, 900*time.Second)257 defer cancel()258259 _, err = client.Models.GenerateContent(260 timeoutCtx,261 "gemini-3.8-flash",262 genai.Text("why is the sky blue?"),263 &genai.GenerateContentConfig{264 ServiceTier: "flex",265 },266 )267 if err != nil {268 fmt.Printf("Flex request failed: %v\n", err)269 }270271 // Example with streaming272 streamTimeoutCtx, streamCancel := context.WithTimeout(ctx, 60*time.Second)273 defer streamCancel()274275 iter := client.Models.GenerateContentStream(276 streamTimeoutCtx,277 "gemini-3.8-flash",278 genai.Text("List 5 ideas for a sci-fi movie."),279 &genai.GenerateContentConfig{280 ServiceTier: "flex",281 },282 )283 for {284 response, err := iter.Next()285 if err == iterator.Done {286 break287 }288 if err != nil {289 fmt.Printf("An error occurred during streaming: %v\n", err)290 break291 }292 fmt.Print(response.Candidates[0].Content.Parts[0])293 }294 }295296### REST297298When making REST calls, you can control timeouts using a combination of HTTP299headers and `curl` options:300301- **`X-Server-Timeout` header (server-side timeout)**: This header suggests a302 preferred timeout duration (default 600s) to the Gemini API server. The server303 will attempt to respect this, but it's not guaranteed. The value should be in304 seconds.305306- **`--max-time` in `curl` (Client-Side Timeout)** : The `curl --max-time307 <seconds>` option sets a hard limit on the total time (in seconds) that `curl`308 will wait for the entire operation to complete. This is a client-side safeguard.309310 # Set a server timeout hint of 120 seconds and a client-side curl timeout of 125 seconds.311 curl --max-time 125 \312 -X POST "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.8-flash:generateContent?key=$GEMINI_API_KEY" \313 -H "Content-Type: application/json" \314 -H "X-Server-Timeout: 120" \315 -d '{316 "contents": [{317 "parts":[{"text": "Summarize the latest research on quantum computing."}]318 }],319 "service_tier": "flex"320 }'321322### Global timeouts323324If you want all API calls made through a specific `genai.Client` instance325(client libraries only) to have a default timeout, you can configure this when326initializing the client using `http_options` and `genai.types.HttpOptions`.327328### Python329330 from google import genai331 from google.genai import types332333 global_timeout_ms = 120000334335 client_with_global_timeout = genai.Client(336 http_options=types.HttpOptions(timeout=global_timeout_ms)337 )338339 try:340 # Calling generate_content using global timeout...341 response = client_with_global_timeout.models.generate_content(342 model="gemini-3.8-flash",343 contents="Summarize the history of AI development since 2000.",344 config={"service_tier": "flex"},345 )346 print(response.text)347348 # A per-request timeout will *override* the global timeout for that specific call.349 shorter_timeout = 30000350 response = client_with_global_timeout.models.generate_content(351 model="gemini-3.8-flash",352 contents="Provide a very brief definition of machine learning.",353 config={354 "service_tier": "flex",355 "http_options":{"timeout": shorter_timeout}356 } # Overrides the global timeout357 )358359 print(response.text)360361 except TimeoutError:362 print(363 f"A GenerateContent call timed out. Check if the global or per-request timeout was exceeded."364 )365 except Exception as e:366 print(f"An error occurred: {e}")367368### JavaScript369370 import {GoogleGenAI} from '@google/genai';371372 const globalTimeoutMs = 120000;373374 const clientWithGlobalTimeout = new GoogleGenAI({httpOptions: {timeout: globalTimeoutMs}});375376 async function main() {377 try {378 // Calling generate_content using global timeout...379 const response1 = await clientWithGlobalTimeout.models.generateContent({380 model: "gemini-3.8-flash",381 contents: "Summarize the history of AI development since 2000.",382 config: { serviceTier: "flex" },383 });384 console.log(response1.text());385386 // A per-request timeout will *override* the global timeout for that specific call.387 const shorterTimeout = 30000;388 const response2 = await clientWithGlobalTimeout.models.generateContent({389 model: "gemini-3.8-flash",390 contents: "Provide a very brief definition of machine learning.",391 config: {392 serviceTier: "flex",393 httpOptions: {timeout: shorterTimeout}394 } // Overrides the global timeout395 });396397 console.log(response2.text());398399 } catch (e) {400 if (e.name === 'TimeoutError' || e.message?.includes('timeout')) {401 console.log(402 "A GenerateContent call timed out. Check if the global or per-request timeout was exceeded."403 );404 } else {405 console.log(`An error occurred: ${e}`);406 }407 }408 }409410 await main();411412### Go413414 package main415416 import (417 "context"418 "fmt"419 "log"420 "time"421422 "google.golang.org/genai"423 )424425 func main() {426 ctx := context.Background()427 client, err := genai.NewClient(ctx, nil)428 if err != nil {429 log.Fatal(err)430 }431 defer client.Close()432433 model := client.GenerativeModel("gemini-3.8-flash")434435 // Go uses context for timeouts, not client options.436 // Set a default timeout for requests.437 globalTimeout := 120 * time.Second438 fmt.Printf("Using default timeout of %v seconds.\n", globalTimeout.Seconds())439440 fmt.Println("Calling generate_content (using default timeout)...")441 ctx1, cancel1 := context.WithTimeout(ctx, globalTimeout)442 defer cancel1()443 resp1, err := model.GenerateContent(ctx1, genai.Text("Summarize the history of AI development since 2000."), &genai.GenerateContentConfig{ServiceTier: "flex"})444 if err != nil {445 log.Printf("Request 1 failed: %v", err)446 } else {447 fmt.Println("GenerateContent 1 successful.")448 fmt.Println(resp1.Text())449 }450451 // A different timeout can be used for other requests.452 shorterTimeout := 30 * time.Second453 fmt.Printf("\nCalling generate_content with a shorter timeout of %v seconds...\n", shorterTimeout.Seconds())454 ctx2, cancel2 := context.WithTimeout(ctx, shorterTimeout)455 defer cancel2()456 resp2, err := model.GenerateContent(ctx2, genai.Text("Provide a very brief definition of machine learning."), &genai.GenerateContentConfig{457 ServiceTier: "flex",458 })459 if err != nil {460 log.Printf("Request 2 failed: %v", err)461 } else {462 fmt.Println("GenerateContent 2 successful.")463 fmt.Println(resp2.Text())464 }465 }466467## Implement retries468469Because Flex is sheddable and fails with 503 errors, here is an example of470optionally implementing retry logic to continue with failed requests:471472### Python473474 import time475 from google import genai476477 client = genai.Client()478479 def call_with_retry(max_retries=3, base_delay=5):480 for attempt in range(max_retries):481 try:482 return client.models.generate_content(483 model="gemini-3.8-flash",484 contents="Analyze this batch statement.",485 config={"service_tier": "flex"},486 )487 except Exception as e:488 # Check for 503 Service Unavailable or 429 Rate Limits489 print(e.code)490 if attempt < max_retries - 1:491 delay = base_delay * (2 ** attempt) # Exponential Backoff492 print(f"Flex busy, retrying in {delay}s...")493 time.sleep(delay)494 else:495 # Fallback to standard on last strike (Optional)496 print("Flex exhausted, falling back to Standard...")497 return client.models.generate_content(498 model="gemini-3.8-flash",499 contents="Analyze this batch statement."500 )501502 # Usage503 response = call_with_retry()504 print(response.text)505506### JavaScript507508 import {GoogleGenAI} from '@google/genai';509510 const ai = new GoogleGenAI({});511512 async function sleep(ms) {513 return new Promise(resolve => setTimeout(resolve, ms));514 }515516 async function callWithRetry(maxRetries = 3, baseDelay = 5) {517 for (let attempt = 0; attempt < maxRetries; attempt++) {518 try {519 console.log(`Attempt ${attempt + 1}: Calling Flex tier...`);520 const response = await ai.models.generateContent({521 model: "gemini-3.8-flash",522 contents: "Analyze this batch statement.",523 config: { serviceTier: 'flex' },524 });525 return response;526 } catch (e) {527 if (attempt < maxRetries - 1) {528 const delay = baseDelay * (2 ** attempt);529 console.log(`Flex busy, retrying in ${delay}s...`);530 await sleep(delay * 1000);531 } else {532 console.log("Flex exhausted, falling back to Standard...");533 return await ai.models.generateContent({534 model: "gemini-3.8-flash",535 contents: "Analyze this batch statement.",536 });537 }538 }539 }540 }541542 async function main() {543 const response = await callWithRetry();544 console.log(response.text);545 }546547 await main();548549### Go550551 package main552553 import (554 "context"555 "fmt"556 "log"557 "math"558 "time"559560 "google.golang.org/genai"561 )562563 func callWithRetry(ctx context.Context, client *genai.Client, maxRetries int, baseDelay time.Duration) (*genai.GenerateContentResponse, error) {564 modelName := "gemini-3.8-flash"565 content := genai.Text("Analyze this batch statement.")566 flexConfig := &genai.GenerateContentConfig{567 ServiceTier: "flex",568 }569570 for attempt := 0; attempt < maxRetries; attempt++ {571 log.Printf("Attempt %d: Calling Flex tier...", attempt+1)572 resp, err := client.Models.GenerateContent(ctx, modelName, content, flexConfig)573 if err == nil {574 return resp, nil575 }576577 log.Printf("Attempt %d failed: %v", attempt+1, err)578579 if attempt < maxRetries-1 {580 delay := time.Duration(float64(baseDelay) * math.Pow(2, float64(attempt)))581 log.Printf("Flex busy, retrying in %v...", delay)582 time.Sleep(delay)583 } else {584 log.Println("Flex exhausted, falling back to Standard...")585 return client.Models.GenerateContent(ctx, modelName, content)586 }587 }588 return nil, fmt.Errorf("retries exhausted") // Should not be reached589 }590591 func main() {592 ctx := context.Background()593 client, err := genai.NewClient(ctx, nil)594 if err != nil {595 log.Fatal(err)596 }597 defer client.Close()598599 resp, err := callWithRetry(ctx, client, 3, 5*time.Second)600 if err != nil {601 log.Fatalf("Failed after retries: %v", err)602 }603 fmt.Println(resp.Text())604 }605606## Pricing607608Flex inference is priced at 50% of the [standard API](https://ai.google.dev/gemini-api/docs/pricing)609and billed per token.610611## Supported models612613The following models support Flex inference:614615| Model | Flex inference |616|---|---|617| [Gemini 3.8 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash) | ✔️ |618| [Gemini 3.7 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-3.7-flash) | ✔️ |619| [Gemini 3.6 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-3.6-flash) | ✔️ |620| [Gemini 3.5 Flash-Lite](https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash-lite) | ✔️ |621| [Gemini 3.5 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash) | ✔️ |622| [Gemini 3.1 Flash-Lite](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite) | ✔️ |623| [Gemini 3.1 Pro Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview) | ✔️ |624| [Gemini 3 Flash Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3-flash-preview) | ✔️ |625| [Gemini 3 Pro Image Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image) | ✔️ |626| [Gemini 2.5 Pro](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro) | ✔️ |627| [Gemini 2.5 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash) | ✔️ |628| [Gemini 2.5 Flash Image](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-image) | ✔️ |629| [Gemini 2.5 Flash-Lite](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-lite) | ✔️ |630631## What's next632633Read about Gemini's other [inference and optimization](https://ai.google.dev/gemini-api/docs/optimization) options:634635- [Priority inference](https://ai.google.dev/gemini-api/docs/priority-inference) for ultra-low latency.636- [Batch API](https://ai.google.dev/gemini-api/docs/batch-api) for asynchronous processing within 24 hours.637- [Context caching](https://ai.google.dev/gemini-api/docs/caching) for reduced input token costs.