Repository Public
1<br />23Gemini Robotics ER models can point to objects, track them in video, detect4them with bounding boxes, and generate movement trajectories. All examples on5this page use natural language prompts with `generateContent`.67For full runnable code, see the8[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).910## Point to objects1112The following example finds specific objects in an image and returns their13normalized `[y, x]` coordinates:1415### Python1617 from google import genai18 from google.genai import types1920 PROMPT = """21 Point to no more than 10 items in the image. The label returned22 should be an identifying name for the object detected.23 The answer should follow the json format: [{"point": <point>,24 "label": <label1>}, ...]. The points are in [y, x] format25 normalized to 0-1000.26 """27 client = genai.Client()2829 # Load your image30 with open("my-image.png", 'rb') as f:31 image_bytes = f.read()3233 image_response = client.models.generate_content(34 model="gemini-robotics-er-2-preview",35 contents=[36 types.Part.from_bytes(37 data=image_bytes,38 mime_type='image/png',39 ),40 PROMPT41 ],42 config = types.GenerateContentConfig(43 thinking_config=types.ThinkingConfig(thinking_level="high")44 )45 )4647 print(image_response.text)4849### REST5051 # First, ensure you have the image file locally.52 # Encode the image to base6453 IMAGE_BASE64=$(base64 -w 0 my-image.png)5455 curl -X POST \56 "https://generativelanguage.googleapis.com/v1beta/models/gemini-robotics-er-2-preview:generateContent \57 -H "x-goog-api-key: $GEMINI_API_KEY" \58 -H "Content-Type: application/json" \59 -d '{60 "contents": [61 {62 "parts": [63 {64 "inlineData": {65 "mimeType": "image/png",66 "data": "'"${IMAGE_BASE64}"'"67 }68 },69 {70 "text": "Point to no more than 10 items in the image. The label returned should be an identifying name for the object detected. The answer should follow the json format: [{\"point\": [y, x], \"label\": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000."71 }72 ]73 }74 ],75 "generationConfig": {76 "thinkingConfig": {77 "thinkingLevel": "high"78 }79 }80 }'8182The output will be a JSON array containing objects, each with a `point`83(normalized `[y, x]` coordinates) and a `label` identifying the object.8485### JSON8687 [88 {"point": [376, 508], "label": "small banana"},89 {"point": [287, 609], "label": "larger banana"},90 {"point": [223, 303], "label": "pink starfruit"},91 {"point": [435, 172], "label": "paper bag"},92 {"point": [270, 786], "label": "green plastic bowl"},93 {"point": [488, 775], "label": "metal measuring cup"},94 {"point": [673, 580], "label": "dark blue bowl"},95 {"point": [471, 353], "label": "light blue bowl"},96 {"point": [492, 497], "label": "bread"},97 {"point": [525, 429], "label": "lime"}98 ]99100The following image is an example of how these points can be displayed:101102103104## Tracking objects in a video105106Gemini Robotics ER 2 can also analyze video frames to track objects107over time. See [Video inputs](https://ai.google.dev/gemini-api/docs/video-understanding#supported-formats)108for a list of supported video formats.109110### Python111112 from google import genai113 from google.genai import types114115 client = genai.Client()116117 # Load your video118 with open("my-video.mp4", 'rb') as f:119 video_bytes = f.read()120121 prompt = """122 Point to the red ball in every frame where it appears.123 The answer should follow the json format: [{"point": [y, x],124 "label": <label>}, ...]. The points are in [y, x] format125 normalized to 0-1000. Return one entry per frame that contains126 the object.127 """128129 image_response = client.models.generate_content(130 model="gemini-robotics-er-2-preview",131 contents=[132 types.Part.from_bytes(133 data=video_bytes,134 mime_type='video/mp4',135 ),136 prompt137 ],138 config = types.GenerateContentConfig(139 thinking_config=types.ThinkingConfig(thinking_level="high")140 )141 )142143 print(image_response.text)144145## Object detection and bounding boxes146147In addition to points, you can prompt the model to return 2D bounding boxes,148which provide more spatial detail for detected objects.149150### Python151152 from google import genai153 from google.genai import types154155 client = genai.Client()156157 with open("my-image.png", 'rb') as f:158 image_bytes = f.read()159160 prompt = """161 Detect all objects in this image and return bounding boxes.162 The answer should follow the JSON format:163 [{"label": <label>, "y": <y_min>, "x": <x_min>,164 "y2": <y_max>, "x2": <x_max>}, ...]165 where coordinates are normalized to 0-1000.166 """167168 image_response = client.models.generate_content(169 model="gemini-robotics-er-2-preview",170 contents=[171 types.Part.from_bytes(172 data=image_bytes,173 mime_type='image/png',174 ),175 prompt176 ],177 config = types.GenerateContentConfig(178 thinking_config=types.ThinkingConfig(thinking_level="low")179 )180 )181182 print(image_response.text)183184## Trajectories185186Gemini Robotics ER 2 can generate sequences of points that define a187trajectory, useful for guiding robot movement.188189This example requests a trajectory to move a red pen to an organizer, including190an estimate of the intermediate waypoints. The code has been reduced to show191only the prompt.192193### Python194195 prompt = """196 Generate a trajectory for the robotic arm to pick up the red pen197 and place it in the organizer. Return a list of waypoints as JSON:198 [{"step": <int>, "point": [y, x], "action": <description>}, ...]199 where coordinates are normalized to 0-1000.200 """201202## Making room for a laptop203204This example shows how Gemini Robotics ER can reason about a space. The prompt205asks the model to identify which object needs to be moved to create206space for another item.207208### Python209210 from google import genai211 from google.genai import types212213 client = genai.Client()214215 with open('path/to/image-with-objects.jpg', 'rb') as f:216 image_bytes = f.read()217218 prompt = """219 Point to the object that I need to remove to make room for my laptop220 The answer should follow the JSON format: [{"point": <point>,221 "label": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000.222 """223224 image_response = client.models.generate_content(225 model="gemini-robotics-er-2-preview",226 contents=[227 types.Part.from_bytes(228 data=image_bytes,229 mime_type='image/jpeg',230 ),231 prompt232 ],233 config=types.GenerateContentConfig(234 thinking_config=types.ThinkingConfig(thinking_level="high")235 )236 )237238 print(image_response.text)239240The response contains a 2D coordinate of the object that answers the user's241question, in this case, the object that should move to make room for a laptop.242243 [244 {"point": [672, 301], "label": "The object that I need to remove to make room for my laptop"}245 ]246247248249## Packing a lunch250251The model can also provide instructions for multi-step tasks and point to252relevant objects for each step. This example shows how the model plans a series253of steps to pack a lunch bag.254255### Python256257 from google import genai258 from google.genai import types259260 client = genai.Client()261262 with open('path/to/image-of-lunch.jpg', 'rb') as f:263 image_bytes = f.read()264265 prompt = """266 Explain how to pack the lunch box and lunch bag. Point to each267 object that you refer to. Each point should be in the format:268 [{"point": [y, x], "label": }], where the coordinates are269 normalized between 0-1000.270 """271272 image_response = client.models.generate_content(273 model="gemini-robotics-er-2-preview",274 contents=[275 types.Part.from_bytes(276 data=image_bytes,277 mime_type='image/jpeg',278 ),279 prompt280 ],281 config=types.GenerateContentConfig(282 thinking_config=types.ThinkingConfig(thinking_level="high")283 )284 )285286 print(image_response.text)287288The response of this prompt is a set of step by step instructions on how to pack289a lunch bag from the image input.290291**Input image**292293294295**Model output**296297 Based on the image, here is a plan to pack the lunch box and lunch bag:298299 1. **Pack the fruit into the lunch box.** Place the [apple](apple), [banana](banana), [red grapes](red grapes), and [green grapes](green grapes) into the [blue lunch box](blue lunch box).300 2. **Add the spoon to the lunch box.** Put the [blue spoon](blue spoon) inside the lunch box as well.301 3. **Close the lunch box.** Secure the lid on the [blue lunch box](blue lunch box).302 4. **Place the lunch box inside the lunch bag.** Put the closed [blue lunch box](blue lunch box) into the [brown lunch bag](brown lunch bag).303 5. **Pack the remaining items into the lunch bag.** Place the [blue snack bar](blue snack bar) and the [brown snack bar](brown snack bar) into the [brown lunch bag](brown lunch bag).304305 Here is the list of objects and their locations:306 * [{"point": [899, 440], "label": "apple"}]307 * [{"point": [814, 363], "label": "banana"}]308 * [{"point": [727, 470], "label": "red grapes"}]309 * [{"point": [675, 608], "label": "green grapes"}]310 * [{"point": [706, 529], "label": "blue lunch box"}]311 * [{"point": [864, 517], "label": "blue spoon"}]312 * [{"point": [499, 401], "label": "blue snack bar"}]313 * [{"point": [614, 705], "label": "brown snack bar"}]314 * [{"point": [448, 501], "label": "brown lunch bag"}]315316## What's next317318- [Agentic capabilities](https://ai.google.dev/gemini-api/docs/robotics-agentic) --- code execution, instrument reading, image annotation.319- [Task orchestration](https://ai.google.dev/gemini-api/docs/robotics-orchestration) --- long-horizon tasks with custom robot APIs.320- [Robotics with streaming](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time bidirectional streaming (Gemini Robotics ER 2 only).321- [Video understanding](https://ai.google.dev/gemini-api/docs/robotics-video-progress) --- moment finding and progress classification (Gemini Robotics ER 2 only).