Repository Public
1# Image understanding23> [!NOTE]4> **Note** : This version of the page covers the new [Interactions API](https://ai.google.dev/gemini-api/docs/interactions/interactions-overview), which is currently in Beta. 5> For stable production deployments, we recommend you continue to use the `generateContent` API. You can use the toggle on this page to switch between the versions.67Gemini models are built to be multimodal from the ground up, unlocking a wide8range of image processing and computer vision tasks including but not limited to9image captioning, classification, and visual question answering without having10to train specialized ML models.1112In addition to their general multimodal capabilities, Gemini models offer13**enhanced accuracy** for specific use cases like [object detection](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#object-detection) and [segmentation](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#segmentation), through additional14training.1516## Passing images to Gemini1718You can provide images as input to Gemini using several methods:1920- [Passing image using URL](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#url-image): Ideal for publicly accessible images.21- [Passing inline image data](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#inline-image): For base64-encoded image data.22- [Uploading images using the File API](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#upload-image): Recommended for larger files or for reusing images across multiple requests.2324### Passing image using URL2526You can upload an image using the [Files API](https://ai.google.dev/gemini-api/docs/interactions/files) and pass it27in the request:2829### Python3031 from google import genai3233 client = genai.Client()3435 uploaded_file = client.files.upload(file="path/to/organ.jpg")3637 interaction = client.interactions.create(38 model="gemini-3.5-flash",39 input=[40 {"type": "text", "text": "Caption this image."},41 {42 "type": "image",43 "uri": uploaded_file.uri,44 "mime_type": uploaded_file.mime_type45 }46 ]47 )48 print(interaction.output_text)4950### JavaScript5152 import { GoogleGenAI } from "@google/genai";5354 const client = new GoogleGenAI({});5556 const uploadedFile = await client.files.upload({57 file: "path/to/organ.jpg",58 config: { mime_type: "image/jpeg" }59 });6061 const interaction = await client.interactions.create({62 model: "gemini-3.5-flash",63 input: [64 {type: "text", text: "Caption this image."},65 {66 type: "image",67 uri: uploadedFile.uri,68 mime_type: uploadedFile.mimeType69 }70 ]71 });72 console.log(interaction.output_text);7374### REST7576 # First upload the file using the Files API, then use the URI:77 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \78 -H "x-goog-api-key: $GEMINI_API_KEY" \79 -H 'Content-Type: application/json' \80 -H "Api-Revision: 2026-05-20" \81 -d '{82 "model": "gemini-3.5-flash",83 "input": [84 {"type": "text", "text": "Caption this image."},85 {86 "type": "image",87 "uri": "YOUR_FILE_URI",88 "mime_type": "image/jpeg"89 }90 ]91 }'9293### Passing inline image data9495You can provide image data as base64-encoded strings:9697### Python9899 import base64100 from google import genai101102 with open('path/to/small-sample.jpg', 'rb') as f:103 image_bytes = f.read()104105 client = genai.Client()106107 interaction = client.interactions.create(108 model="gemini-3.5-flash",109 input=[110 {"type": "text", "text": "Caption this image."},111 {112 "type": "image",113 "data": base64.b64encode(image_bytes).decode('utf-8'),114 "mime_type": "image/jpeg"115 }116 ]117 )118 print(interaction.output_text)119120### JavaScript121122 import { GoogleGenAI } from "@google/genai";123 import * as fs from "node:fs";124125 const client = new GoogleGenAI({});126 const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {127 encoding: "base64",128 });129130 const interaction = await client.interactions.create({131 model: "gemini-3.5-flash",132 input: [133 {type: "text", text: "Caption this image."},134 {135 type: "image",136 data: base64ImageFile,137 mime_type: "image/jpeg"138 }139 ]140 });141 console.log(interaction.output_text);142143### REST144145 IMG_PATH="/path/to/your/image1.jpg"146147 if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then148 B64FLAGS="--input"149 else150 B64FLAGS="-w0"151 fi152153 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \154 -H "x-goog-api-key: $GEMINI_API_KEY" \155 -H 'Content-Type: application/json' \156 -H "Api-Revision: 2026-05-20" \157 -d '{158 "model": "gemini-3.5-flash",159 "input": [160 {"type": "text", "text": "Caption this image."},161 {162 "type": "image",163 "data": "'"$(base64 $B64FLAGS $IMG_PATH)"'",164 "mime_type": "image/jpeg"165 }166 ]167 }'168169> [!NOTE]170> **Note:** Inline image data limits your total request size (text prompts, system instructions, and inline bytes) to 20MB. For larger requests, [upload image files](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#upload-image) using the File API.171172### Uploading images using the File API173174For large files or to be able to use the same image file repeatedly, use the175Files API. See the [Files API guide](https://ai.google.dev/gemini-api/docs/interactions/files).176177### Python178179 from google import genai180181 client = genai.Client()182183 my_file = client.files.upload(file="path/to/sample.jpg")184185 interaction = client.interactions.create(186 model="gemini-3.5-flash",187 input=[188 {"type": "text", "text": "Caption this image."},189 {190 "type": "image",191 "uri": my_file.uri,192 "mime_type": my_file.mime_type193 }194 ]195 )196 print(interaction.output_text)197198### JavaScript199200 import { GoogleGenAI } from "@google/genai";201202 const client = new GoogleGenAI({});203204 const myfile = await client.files.upload({205 file: "path/to/sample.jpg",206 config: { mimeType: "image/jpeg" },207 });208209 const interaction = await client.interactions.create({210 model: "gemini-3.5-flash",211 input: [212 {type: "text", text: "Caption this image."},213 {214 type: "image",215 uri: myfile.uri,216 mime_type: myfile.mimeType217 }218 ]219 });220 console.log(interaction.output_text);221222### REST223224 # First upload the file (see Files API guide for details)225 # Then use the file URI in the request:226227 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \228 -H "x-goog-api-key: $GEMINI_API_KEY" \229 -H 'Content-Type: application/json' \230 -H "Api-Revision: 2026-05-20" \231 -d '{232 "model": "gemini-3.5-flash",233 "input": [234 {"type": "text", "text": "Caption this image."},235 {236 "type": "image",237 "uri": "YOUR_FILE_URI",238 "mime_type": "image/jpeg"239 }240 ]241 }'242243## Prompting with multiple images244245You can provide multiple images in a single prompt by including multiple image246objects in the `input` array:247248### Python249250 from google import genai251252 client = genai.Client()253254 interaction = client.interactions.create(255 model="gemini-3.5-flash",256 input=[257 {"type": "text", "text": "What is different between these two images?"},258 {259 "type": "image",260 "uri": "https://example.com/image1.jpg",261 "mime_type": "image/jpeg"262 },263 {264 "type": "image",265 "uri": "https://example.com/image2.jpg",266 "mime_type": "image/jpeg"267 }268 ]269 )270 print(interaction.output_text)271272### JavaScript273274 import { GoogleGenAI } from "@google/genai";275276 const client = new GoogleGenAI({});277278 const interaction = await client.interactions.create({279 model: "gemini-3.5-flash",280 input: [281 {type: "text", text: "What is different between these two images?"},282 {283 type: "image",284 uri: "https://example.com/image1.jpg",285 mime_type: "image/jpeg"286 },287 {288 type: "image",289 uri: "https://example.com/image2.jpg",290 mime_type: "image/jpeg"291 }292 ]293 });294 console.log(interaction.output_text);295296### REST297298 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \299 -H "x-goog-api-key: $GEMINI_API_KEY" \300 -H 'Content-Type: application/json' \301 -H "Api-Revision: 2026-05-20" \302 -d '{303 "model": "gemini-3.5-flash",304 "input": [305 {"type": "text", "text": "What is different between these two images?"},306 {307 "type": "image",308 "uri": "https://example.com/image1.jpg",309 "mime_type": "image/jpeg"310 },311 {312 "type": "image",313 "uri": "https://example.com/image2.jpg",314 "mime_type": "image/jpeg"315 }316 ]317 }'318319## Object detection320321Models are trained to detect objects in an322image and get their bounding box coordinates. The coordinates, relative to image323dimensions, scale to \[0, 1000\]. You need to descale these coordinates based on324your original image size.325326### Python327328 from google import genai329 from pydantic import BaseModel, Field330 from typing import List331 import json332333 client = genai.Client()334 prompt = "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000."335336 class BoundingBox(BaseModel):337 box_2d: List[int] = Field(description="The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000.")338 mask: List[List[int]] = Field(description="The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000.")339 label: str = Field(description="A descriptive label for the item.")340341 class BoundingBoxes(BaseModel):342 boxes: List[BoundingBox]343344 interaction = client.interactions.create(345 model="gemini-3.5-flash",346 input=[347 {"type": "text", "text": prompt},348 {349 "type": "image",350 "uri": "https://example.com/image.png",351 "mime_type": "image/png"352 }353 ],354 response_format={355 "type": "text",356 "mime_type": "application/json",357 "schema": BoundingBoxes.model_json_schema()358 }359 )360361 bounding_boxes = BoundingBoxes.model_validate_json(interaction.output_text)362 print(bounding_boxes)363364### JavaScript365366 import { GoogleGenAI } from "@google/genai";367 import * as z from "zod";368369 const client = new GoogleGenAI({});370 const prompt = "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000.";371372 const boundingBoxesSchema = z.object({373 boxes: z.array(z.object({374 box_2d: z.array(z.number()),375 mask: z.array(z.array(z.number())),376 label: z.string()377 }))378 });379380 const interaction = await client.interactions.create({381 model: "gemini-3.5-flash",382 input: [383 { type: "text", text: prompt },384 {385 type: "image",386 uri: "https://example.com/image.png",387 mime_type: "image/png"388 }389 ],390 response_format: {391 type: 'text',392 mime_type: 'application/json',393 schema: z.toJSONSchema(boundingBoxesSchema)394 },395 });396397 const result = boundingBoxesSchema.parse(JSON.parse(interaction.output_text));398 console.log(result);399400### REST401402 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \403 -H "x-goog-api-key: $GEMINI_API_KEY" \404 -H 'Content-Type: application/json' \405 -H "Api-Revision: 2026-05-20" \406 -d '{407 "model": "gemini-3.5-flash",408 "input": [409 {"type": "text", "text": "Detect the all of the prominent items in the image. The box_2d should be [ymin, xmin, ymax, xmax] normalized to 0-1000."},410 {411 "type": "image",412 "uri": "https://example.com/image.png",413 "mime_type": "image/png"414 }415 ],416 "response_format": {417 "type": "text",418 "mime_type": "application/json",419 "schema": {420 "type": "object",421 "properties": {422 "boxes": {423 "type": "array",424 "items": {425 "type": "object",426 "properties": {427 "box_2d": { "type": "array", "items": { "type": "integer" } },428 "mask": { "type": "array", "items": { "type": "array", "items": { "type": "integer" } } },429 "label": { "type": "string" }430 },431 "required": ["box_2d", "mask", "label"]432 }433 }434 },435 "required": ["boxes"]436 }437 }438 }'439440> [!NOTE]441> **Note:** The model also supports generating bounding boxes based on custom instructions, such as: "Show bounding boxes of all green objects in this image".442443For more examples, check following notebooks in the [Gemini Cookbook](https://github.com/google-gemini/cookbook):444445## Segmentation446447Starting with Gemini 2.5, models not only detect items but also segment them448and provide their contour masks.449450The model predicts a JSON list, where each item represents a segmentation mask.451Each item has a bounding box ("`box_2d`") in the format `[y0, x0, y1, x1]` with452normalized coordinates between 0 and 1000, a label ("`label`") that identifies453the object, and finally the segmentation mask inside the bounding box, as base64454encoded png that is a probability map with values between 0 and 255.455456> [!NOTE]457> **Note:** For better results, disable [thinking](https://ai.google.dev/gemini-api/docs/interactions/thinking) by setting the thinking level to "minimal".458459### Python460461 from google import genai462 from pydantic import BaseModel, Field463 from typing import List464 import json465466 client = genai.Client()467468 prompt = """469 Give the segmentation masks for the wooden and glass items.470 Output a JSON list of segmentation masks where each entry contains the 2D471 bounding box in the key "box_2d", the segmentation mask in key "mask", and472 the text label in the key "label". Use descriptive labels.473 """474475 class BoundingBox(BaseModel):476 box_2d: List[int] = Field(description="The 2D bounding box of the item as [ymin, xmin, ymax, xmax] normalized to 0-1000.")477 mask: List[List[int]] = Field(description="The segmentation mask of the item as a polygon of [x,y] coordinates, normalized to 0-1000.")478 label: str = Field(description="A descriptive label for the item.")479480 class BoundingBoxes(BaseModel):481 boxes: List[BoundingBox]482483 interaction = client.interactions.create(484 model="gemini-3.5-flash",485 input=[486 {"type": "text", "text": prompt},487 {488 "type": "image",489 "uri": "https://example.com/image.png",490 "mime_type": "image/png"491 }492 ],493 response_format={494 "type": "text",495 "mime_type": "application/json",496 "schema": BoundingBoxes.model_json_schema()497 },498 generation_config={499 "thinking_level": "minimal"500 }501 )502503 items = BoundingBoxes.model_validate_json(interaction.output_text)504 print("Segmentation results:", items)505506### JavaScript507508 import { GoogleGenAI } from "@google/genai";509 import * as z from "zod";510511 const client = new GoogleGenAI({});512 const prompt = `513 Give the segmentation masks for the wooden and glass items.514 Output a JSON list of segmentation masks where each entry contains the 2D515 bounding box in the key "box_2d", the segmentation mask in key "mask", and516 the text label in the key "label". Use descriptive labels.517 `;518519 const boundingBoxesSchema = z.object({520 boxes: z.array(z.object({521 box_2d: z.array(z.number()),522 mask: z.array(z.array(z.number())),523 label: z.string()524 }))525 });526527 const interaction = await client.interactions.create({528 model: "gemini-3.5-flash",529 input: [530 { type: "text", text: prompt },531 {532 type: "image",533 uri: "https://example.com/image.png",534 mime_type: "image/png"535 }536 ],537 response_format: {538 type: 'text',539 mime_type: 'application/json',540 schema: z.toJSONSchema(boundingBoxesSchema)541 },542 generation_config: {543 thinking_level: "minimal"544 }545 });546547 const result = boundingBoxesSchema.parse(JSON.parse(interaction.output_text));548 console.log(result);549550### REST551552 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \553 -H "x-goog-api-key: $GEMINI_API_KEY" \554 -H 'Content-Type: application/json' \555 -H "Api-Revision: 2026-05-20" \556 -d '{557 "model": "gemini-3.5-flash",558 "input": [559 {"type": "text", "text": "Give the segmentation masks for the wooden and glass items.\nOutput a JSON list of segmentation masks where each entry contains the 2D\nbounding box in the key \"box_2d\", the segmentation mask in key \"mask\", and\nthe text label in the key \"label\". Use descriptive labels."},560 {561 "type": "image",562 "uri": "https://example.com/image.png",563 "mime_type": "image/png"564 }565 ],566 "response_format": {567 "type": "text",568 "mime_type": "application/json",569 "schema": {570 "type": "object",571 "properties": {572 "boxes": {573 "type": "array",574 "items": {575 "type": "object",576 "properties": {577 "box_2d": { "type": "array", "items": { "type": "integer" } },578 "mask": { "type": "array", "items": { "type": "array", "items": { "type": "integer" } } },579 "label": { "type": "string" }580 },581 "required": ["box_2d", "mask", "label"]582 }583 }584 },585 "required": ["boxes"]586 }587 },588 "generation_config": {589 "thinking_level": "minimal"590 }591 }'592593 An example segmentation output with objects and segmentation masks594595## Supported image formats596597Gemini supports the following image format MIME types:598599- PNG - `image/png`600- JPEG - `image/jpeg`601- WEBP - `image/webp`602- HEIC - `image/heic`603- HEIF - `image/heif`604605To learn about other file input methods, see the606[File input methods](https://ai.google.dev/gemini-api/docs/interactions/file-input-methods) guide.607608## Capabilities609610All Gemini model versions are multimodal and can be utilized in a wide range611of image processing and computer vision tasks including but not limited to612image captioning, visual question and answering, image classification,613object detection and segmentation.614615Gemini can reduce the need to use specialized ML models depending on your616quality and performance requirements.617618The latest model versions are specifically trained improve accuracy of619specialized tasks in addition to generic capabilities, like enhanced620[object detection](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#object-detection) and [segmentation](https://ai.google.dev/gemini-api/docs/interactions/image-understanding#segmentation).621622## Limitations and key technical information623624### File limit625626Gemini models support a maximum of 3,600 image files per request.627628### Token calculation629630- 258 tokens if both dimensions \<= 384 pixels. Larger images are tiled into 768x768 pixel tiles, each costing 258 tokens.631632A rough formula for calculating the number of tiles is as follows:633634- Calculate the crop unit size which is roughly: `floor(min(width, height)` / 1.5).635- Divide each dimension by the crop unit size and multiply together to get the number of tiles.636637For example, for an image of dimensions 960x540 would have a crop unit size638of 360. Divide each dimension by 360 and the number of tile is 3 \* 2 = 6.639640### Media resolution641642Gemini 3 introduces granular control over multimodal vision processing with the643`media_resolution` parameter. The `media_resolution` parameter determines the644**maximum number of tokens allocated per input image or video frame.**645Higher resolutions improve the model's ability to646read fine text or identify small details, but increase token usage and latency.647648## Tips and best practices649650- Verify that images are correctly rotated.651- Use clear, non-blurry images.652- When using a single image with text, place the text prompt *before* the image in the `input` array.653654## What's next655656This guide shows you how to upload image files and generate text outputs657from image inputs. To learn more, see the following resources:658659- [Files API](https://ai.google.dev/gemini-api/docs/interactions/files): Learn more about uploading and managing files for use with Gemini.660- [System instructions](https://ai.google.dev/gemini-api/docs/interactions/text-generation#system-instructions): System instructions let you steer the behavior of the model based on your specific needs and use cases.661- [File prompting strategies](https://ai.google.dev/gemini-api/docs/interactions/files#prompt-guide): The Gemini API supports prompting with text, image, audio, and video data, also known as multimodal prompting.662- [Safety guidance](https://ai.google.dev/gemini-api/docs/safety-guidance): Sometimes generative AI models produce unexpected outputs, such as outputs that are inaccurate, biased, or offensive. Post-processing and human evaluation are essential to limit the risk of harm from such outputs.