Repository Public
1<br />23Gemini Robotics ER models can write and execute Python code to manipulate4images and apply logic before answering. This page covers code execution5examples: object detection with zoom and crop, instrument reading, fluid6measurement, circuit board reading, and image annotation.78To adapt these examples to your own use case, replace the prompt text and the9uploaded image file with your own. You can also adjust the requested JSON10schema in the prompt to match the output structure your application needs, or11add a `system_instruction` to enforce output format and precision.1213For full runnable code, see the14[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).1516## Thinking level1718You can control the thinking level of the model to trade latency for accuracy. Spatial19tasks like object detection perform well with a low thinking level. Complex tasks like20counting or weight estimation benefit from a higher thinking level.2122The following example sets the thinking level to `high` for a complex counting task:2324### Python2526 from google import genai2728 client = genai.Client()2930 uploaded_file = client.files.upload(file="scene.jpeg")3132 interaction = client.interactions.create(33 model="gemini-robotics-er-2-preview",34 input=[35 {36 "type": "image",37 "uri": uploaded_file.uri,38 "mime_type": uploaded_file.mime_type39 },40 {"type": "text", "text": "Identify and count all objects on the table."}41 ],42 generation_config={43 "thinking_level": "high" # Use "minimal" or "low" for faster spatial tasks44 }45 )4647 print(interaction.output_text)4849See [Thinking](https://ai.google.dev/gemini-api/docs/thinking) for details.5051## Object detection (Zoom and crop)5253The following example uses code execution to zoom and crop an54image for a clearer view when detecting objects and returning bounding boxes.5556### Python5758 from google import genai5960 client = genai.Client()6162 uploaded_file = client.files.upload(file="sorting.jpeg")6364 prompt = """65 Return JSON in the format {label: val, y: val, x: val, y2: val, x2: val} for66 the compostable objects in this scene. Please Zoom and crop the image for a67 clearer view. Return an annotated image of the final result with the bounding68 boxes drawn on it to the API caller as a part of your process.69 """7071 interaction = client.interactions.create(72 model="gemini-robotics-er-2-preview",73 input=[74 {75 "type": "image",76 "uri": uploaded_file.uri,77 "mime_type": uploaded_file.mime_type78 },79 {"type": "text", "text": prompt}80 ],81 tools=[{"type": "code_execution"}]82 )8384 print(interaction.output_text)8586The model output would be similar to the following json response:8788 [89 {"label": "compostable", "y": 256, "x": 482, "y2": 295, "x2": 546},90 {"label": "compostable", "y": 317, "x": 478, "y2": 350, "x2": 542},91 {"label": "compostable", "y": 586, "x": 556, "y2": 668, "x2": 595},92 {"label": "compostable", "y": 463, "x": 669, "y2": 511, "x2": 718},93 {"label": "compostable", "y": 178, "x": 565, "y2": 250, "x2": 609}94 ]9596The following image displays the boxes returned from the model.979899100## Read an analog gauge and apply logic101102The following example demonstrates how to use the model to read an analog gauge103and perform time calculations. It uses a system instruction to enforce a JSON output.104105### Python106107 from google import genai108109 client = genai.Client()110111 uploaded_file = client.files.upload(file="gauge.jpeg")112113 interaction = client.interactions.create(114 model="gemini-robotics-er-2-preview",115 system_instruction="Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block).",116 input=[117 {118 "type": "image",119 "uri": uploaded_file.uri,120 "mime_type": uploaded_file.mime_type121 },122 {"type": "text", "text": """Read the current value from this gauge. Then, calculate how long123 it will take at the current rate for the value to reach maximum.124 Reply in JSON: {"current_value": val, "max_value": val,125 "time_to_max_minutes": val}"""}126 ],127 tools=[{"type": "code_execution"}]128 )129130 print(interaction.output_text)131132## Measure fluid in a container133134The following example demonstrates how to use code execution to measure the135level of fluid in a container.136137### Python138139 from google import genai140141 client = genai.Client()142143 uploaded_file = client.files.upload(file="fluid.jpeg")144145 interaction = client.interactions.create(146 model="gemini-robotics-er-2-preview",147 system_instruction="Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block).",148 input=[149 {150 "type": "image",151 "uri": uploaded_file.uri,152 "mime_type": uploaded_file.mime_type153 },154 {"type": "text", "text": """Measure the amount of fluid in the container. Reply in JSON:155 {"fluid_level_ml": val, "container_capacity_ml": val,156 "percentage_full": val}"""}157 ],158 tools=[{"type": "code_execution"}]159 )160161 print(interaction.output_text)162163## Read markings on a circuit board164165The following example demonstrates how to use code execution to read the166markings on a circuit board.167168### Python169170 from google import genai171172 client = genai.Client()173174 uploaded_file = client.files.upload(file="circuit_board.jpeg")175176 interaction = client.interactions.create(177 model="gemini-robotics-er-2-preview",178 system_instruction="Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block).",179 input=[180 {181 "type": "image",182 "uri": uploaded_file.uri,183 "mime_type": uploaded_file.mime_type184 },185 {"type": "text", "text": """Read all visible component labels and markings on this circuit186 board. Reply in JSON: {"components": [{"label": val,187 "location": [y, x]}]}"""}188 ],189 tools=[{"type": "code_execution"}]190 )191192 print(interaction.output_text)193194195196## Image annotation197198The following example demonstrates how to use code execution to annotate an199image (e.g., drawing arrows for disposal instructions) and return the modified image.200201### Python202203 from google import genai204205 client = genai.Client()206207 # Load your image208 uploaded_file = client.files.upload(file="sorting.jpeg")209210 prompt = """211 Look at this image and return it as an annotated version using arrows of212 different colors to represent which items should go in which bins for213 disposal. You must return the final image to the API caller.214 """215216 interaction = client.interactions.create(217 model="gemini-robotics-er-2-preview",218 input=[219 {220 "type": "image",221 "uri": uploaded_file.uri,222 "mime_type": uploaded_file.mime_type223 },224 {"type": "text", "text": prompt}225 ],226 tools=[{"type": "code_execution"}]227 )228229 print(interaction.output_text)230231The following is an example image input.232233234235The model output would be similar to the following:236237 The annotated image shows the suggested disposal locations for the items on the table:238 - **Green bin (Compost/Organic)**: Green chili, red chili, grapes, and cherries.239 - **Blue bin (Recycling)**: Yellow crushed can and plastic container.240 - **Black bin (Trash)**: Chocolate bar wrapper, Welch's packet, and white tissue.241242## What's next243244- [Task orchestration](https://ai.google.dev/gemini-api/docs/robotics-orchestration) --- long-horizon tasks with custom robot APIs.245- [Robotics with streaming](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time bidirectional streaming (Gemini Robotics ER 2 only).246- [Video understanding](https://ai.google.dev/gemini-api/docs/robotics-video-progress) --- moment finding and progress classification (Gemini Robotics ER 2 only).