Repository Public
1<br />23Gemini Robotics ER models can point to objects, track them in video, detect4them with bounding boxes, and generate movement trajectories.56For full runnable code, see the7[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).89## Point to objects1011The following example finds specific objects in an image and returns their12normalized `[y, x]` coordinates:1314### Python1516 from google import genai1718 PROMPT = """19 Point to no more than 10 items in the image. The label returned20 should be an identifying name for the object detected.21 The answer should follow the json format: [{"point": <point>,22 "label": <label1>}, ...]. The points are in [y, x] format23 normalized to 0-1000.24 """25 client = genai.Client()2627 uploaded_file = client.files.upload(file="my-image.png")2829 image_response = client.interactions.create(30 model="gemini-robotics-er-2-preview",31 input=[32 {33 "type": "image",34 "uri": uploaded_file.uri,35 "mime_type": uploaded_file.mime_type36 },37 {"type": "text", "text": PROMPT}38 ],39 generation_config={"thinking_level": "high"},40 )4142 print(image_response.output_text)4344### REST4546 # First, ensure you have the image file locally.47 # Encode the image to base6448 IMAGE_BASE64=$(base64 -w 0 my-image.png)4950 curl -X POST \51 "https://generativelanguage.googleapis.com/v1beta/interactions" \52 -H "x-goog-api-key: $GEMINI_API_KEY" \53 -H "Content-Type: application/json" \54 -d '{55 "model": "gemini-robotics-er-2-preview",56 "input": {57 "parts": [58 {59 "inlineData": {60 "mimeType": "image/png",61 "data": "'"${IMAGE_BASE64}"'"62 }63 },64 {65 "text": "Point to no more than 10 items in the image. The label returned should be an identifying name for the object detected. The answer should follow the json format: [{\"point\": [y, x], \"label\": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000."66 }67 ]68 },69 "generation_config": {70 "thinking_config": {71 "thinking_level": "high"72 }73 }74 }'7576The output will be a JSON array containing objects, each with a `point`77(normalized `[y, x]` coordinates) and a `label` identifying the object.7879### JSON8081 [82 {"point": [376, 508], "label": "small banana"},83 {"point": [287, 609], "label": "larger banana"},84 {"point": [223, 303], "label": "pink starfruit"},85 {"point": [435, 172], "label": "paper bag"},86 {"point": [270, 786], "label": "green plastic bowl"},87 {"point": [488, 775], "label": "metal measuring cup"},88 {"point": [673, 580], "label": "dark blue bowl"},89 {"point": [471, 353], "label": "light blue bowl"},90 {"point": [492, 497], "label": "bread"},91 {"point": [525, 429], "label": "lime"}92 ]9394The following image is an example of how these points can be displayed:95969798## Tracking objects in a video99100Gemini Robotics ER 2 can also analyze video frames to track objects101over time. See [Video inputs](https://ai.google.dev/gemini-api/docs/video-understanding#supported-formats)102for a list of supported video formats.103104### Python105106 from google import genai107108 client = genai.Client()109110 uploaded_file = client.files.upload(file="my-video.mp4")111112 prompt = """113 Point to the red ball in every frame where it appears.114 The answer should follow the json format: [{"point": [y, x],115 "label": <label>}, ...]. The points are in [y, x] format116 normalized to 0-1000. Return one entry per frame that contains117 the object.118 """119120 image_response = client.interactions.create(121 model="gemini-robotics-er-2-preview",122 input=[123 {124 "type": "video",125 "uri": uploaded_file.uri,126 "mime_type": uploaded_file.mime_type127 },128 {"type": "text", "text": prompt}129 ],130 )131132 print(image_response.output_text)133134## Object detection and bounding boxes135136In addition to points, you can prompt the model to return 2D bounding boxes,137which provide more spatial detail for detected objects.138139### Python140141 from google import genai142143 client = genai.Client()144145 uploaded_file = client.files.upload(file="my-image.png")146147 prompt = """148 Detect all objects in this image and return bounding boxes.149 The answer should follow the JSON format:150 [{"label": <label>, "y": <y_min>, "x": <x_min>,151 "y2": <y_max>, "x2": <x_max>}, ...]152 where coordinates are normalized to 0-1000.153 """154155 image_response = client.interactions.create(156 model="gemini-robotics-er-2-preview",157 input=[158 {159 "type": "image",160 "uri": uploaded_file.uri,161 "mime_type": uploaded_file.mime_type162 },163 {"type": "text", "text": prompt}164 ],165 )166167 print(image_response.output_text)168169## Trajectories170171Gemini Robotics ER 2 can generate sequences of points that define a172trajectory, useful for guiding robot movement.173174This example requests a trajectory to move a red pen to an organizer, including175an estimate of the intermediate waypoints. The code has been reduced to show176only the prompt.177178### Python179180 prompt = """181 Generate a trajectory for the robotic arm to pick up the red pen182 and place it in the organizer. Return a list of waypoints as JSON:183 [{"step": <int>, "point": [y, x], "action": <description>}, ...]184 where coordinates are normalized to 0-1000.185 """186187## Making room for a laptop188189This example shows how Gemini Robotics ER can reason about a space. The prompt190asks the model to identify which object needs to be moved to create191space for another item.192193### Python194195 from google import genai196197 client = genai.Client()198199 uploaded_file = client.files.upload(file="path/to/image-with-objects.jpg")200201 prompt = """202 Point to the object that I need to remove to make room for my laptop203 The answer should follow the JSON format: [{"point": <point>,204 "label": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000.205 """206207 image_response = client.interactions.create(208 model="gemini-robotics-er-2-preview",209 input=[210 {211 "type": "image",212 "uri": uploaded_file.uri,213 "mime_type": uploaded_file.mime_type214 },215 {"type": "text", "text": prompt}216 ],217 )218219 print(image_response.output_text)220221The response contains a 2D coordinate of the object that answers the user's222question, in this case, the object that should move to make room for a laptop.223224 [225 {"point": [672, 301], "label": "The object that I need to remove to make room for my laptop"}226 ]227228229230## Packing a lunch231232The model can also provide instructions for multi-step tasks and point to233relevant objects for each step. This example shows how the model plans a series234of steps to pack a lunch bag.235236### Python237238 from google import genai239240 client = genai.Client()241242 uploaded_file = client.files.upload(file="path/to/image-of-lunch.jpg")243244 prompt = """245 Explain how to pack the lunch box and lunch bag. Point to each246 object that you refer to. Each point should be in the format:247 [{"point": [y, x], "label": }], where the coordinates are248 normalized between 0-1000.249 """250251 image_response = client.interactions.create(252 model="gemini-robotics-er-2-preview",253 input=[254 {255 "type": "image",256 "uri": uploaded_file.uri,257 "mime_type": uploaded_file.mime_type258 },259 {"type": "text", "text": prompt}260 ],261 )262263 print(image_response.output_text)264265The response of this prompt is a set of step by step instructions on how to pack266a lunch bag from the image input.267268**Input image**269270271272**Model output**273274 Based on the image, here is a plan to pack the lunch box and lunch bag:275276 1. **Pack the fruit into the lunch box.** Place the [apple](apple), [banana](banana), [red grapes](red grapes), and [green grapes](green grapes) into the [blue lunch box](blue lunch box).277 2. **Add the spoon to the lunch box.** Put the [blue spoon](blue spoon) inside the lunch box as well.278 3. **Close the lunch box.** Secure the lid on the [blue lunch box](blue lunch box).279 4. **Place the lunch box inside the lunch bag.** Put the closed [blue lunch box](blue lunch box) into the [brown lunch bag](brown lunch bag).280 5. **Pack the remaining items into the lunch bag.** Place the [blue snack bar](blue snack bar) and the [brown snack bar](brown snack bar) into the [brown lunch bag](brown lunch bag).281282 Here is the list of objects and their locations:283 * [{"point": [899, 440], "label": "apple"}]284 * [{"point": [814, 363], "label": "banana"}]285 * [{"point": [727, 470], "label": "red grapes"}]286 * [{"point": [675, 608], "label": "green grapes"}]287 * [{"point": [706, 529], "label": "blue lunch box"}]288 * [{"point": [864, 517], "label": "blue spoon"}]289 * [{"point": [499, 401], "label": "blue snack bar"}]290 * [{"point": [614, 705], "label": "brown snack bar"}]291 * [{"point": [448, 501], "label": "brown lunch bag"}]292293## What's next294295- [Agentic capabilities](https://ai.google.dev/gemini-api/docs/robotics-agentic) --- code execution, instrument reading, image annotation.296- [Task orchestration](https://ai.google.dev/gemini-api/docs/robotics-orchestration) --- long-horizon tasks with custom robot APIs.297- [Robotics with streaming](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time bidirectional streaming (Gemini Robotics ER 2 only).298- [Video understanding](https://ai.google.dev/gemini-api/docs/robotics-video-progress) --- moment finding and progress classification (Gemini Robotics ER 2 only).