Repository Public
1<br />23Gemini Robotics ER models can write and execute Python code to manipulate4images and apply logic before answering. This page covers code execution5examples: object detection with zoom and crop, instrument reading, fluid6measurement, circuit board reading, and image annotation.78To adapt these examples to your own use case, replace the prompt text and the9uploaded image file with your own. You can also adjust the requested JSON10schema in the prompt to match the output structure your application needs, or11add a `system_instruction` to enforce output format and precision.1213For full runnable code, see the14[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).1516## Thinking level1718You can control the thinking level to trade latency for accuracy. Spatial19tasks like object detection perform well with a low thinking level. Complex tasks like20counting or weight estimation benefit from a higher thinking level.2122The following example sets the thinking level to `high` for a complex counting task:2324### Python2526 from google import genai27 from google.genai import types2829 client = genai.Client()3031 with open('scene.jpeg', 'rb') as f:32 image_bytes = f.read()3334 response = client.models.generate_content(35 model="gemini-robotics-er-2-preview",36 contents=[37 types.Part.from_bytes(38 data=image_bytes,39 mime_type='image/jpeg',40 ),41 "Identify and count all objects on the table."42 ],43 config=types.GenerateContentConfig(44 thinking_config=types.ThinkingConfig(thinking_level="high")45 )46 )4748 print(response.text)4950See [Thinking](https://ai.google.dev/gemini-api/docs/generate-content/thinking) for details.5152## Object detection (Zoom and crop)5354The following example demonstrates how to use code execution to zoom and crop an55image for a clearer view when detecting objects and returning bounding boxes.5657### Python5859 from google import genai60 from google.genai import types6162 client = genai.Client()6364 # Load your image65 with open('sorting.jpeg', 'rb') as f:66 image_bytes = f.read()6768 prompt = """69 Return JSON in the format {label: val, y: val, x: val, y2: val, x2: val} for70 the compostable objects in this scene. Please Zoom and crop the image for a71 clearer view. Return an annotated image of the final result with the bounding72 boxes drawn on it to the API caller as a part of your process.73 """7475 response = client.models.generate_content(76 model="gemini-robotics-er-2-preview",77 contents=[78 types.Part.from_bytes(79 data=image_bytes,80 mime_type='image/jpeg',81 ),82 prompt83 ],84 config = types.GenerateContentConfig(85 tools=[types.Tool(code_execution=types.ToolCodeExecution)],86 )87 )8889 print(response.text)9091The model output would be similar to the following json response:9293 [94 {"label": "compostable", "y": 256, "x": 482, "y2": 295, "x2": 546},95 {"label": "compostable", "y": 317, "x": 478, "y2": 350, "x2": 542},96 {"label": "compostable", "y": 586, "x": 556, "y2": 668, "x2": 595},97 {"label": "compostable", "y": 463, "x": 669, "y2": 511, "x2": 718},98 {"label": "compostable", "y": 178, "x": 565, "y2": 250, "x2": 609}99 ]100101The following image displays the boxes returned from the model.102103104105## Read an analog gauge and apply logic106107The following example demonstrates how to use the model to read an analog gauge108and perform time calculations. It uses a system instruction to enforce a JSON output.109110### Python111112 from google import genai113 from google.genai import types114115 client = genai.Client()116117 with open('gauge.jpeg', 'rb') as f:118 image_bytes = f.read()119120 system_instruction = "Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block)."121122 response = client.models.generate_content(123 model="gemini-robotics-er-2-preview",124 contents=[125 types.Part.from_bytes(126 data=image_bytes,127 mime_type='image/jpeg',128 ),129 """Read the current value from this gauge. Then, calculate how long130 it will take at the current rate for the value to reach maximum.131 Reply in JSON: {"current_value": val, "max_value": val,132 "time_to_max_minutes": val}"""133 ],134 config = types.GenerateContentConfig(135 system_instruction=system_instruction,136 tools=[types.Tool(code_execution=types.ToolCodeExecution)],137 )138 )139140 print(response.text)141142## Measure fluid in a container143144The following example demonstrates how to use code execution to measure the145level of fluid in a container.146147### Python148149 from google import genai150 from google.genai import types151152 client = genai.Client()153154 with open('fluid.jpeg', 'rb') as f:155 image_bytes = f.read()156157 system_instruction = "Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block)."158159 response = client.models.generate_content(160 model="gemini-robotics-er-2-preview",161 contents=[162 types.Part.from_bytes(163 data=image_bytes,164 mime_type='image/jpeg',165 ),166 """Measure the amount of fluid in the container. Reply in JSON:167 {"fluid_level_ml": val, "container_capacity_ml": val,168 "percentage_full": val}"""169 ],170 config = types.GenerateContentConfig(171 system_instruction=system_instruction,172 tools=[types.Tool(code_execution=types.ToolCodeExecution)],173 )174 )175176 print(response.text)177178## Read markings on a circuit board179180The following example demonstrates how to use code execution to read the181markings on a circuit board.182183### Python184185 from google import genai186 from google.genai import types187188 client = genai.Client()189190 with open('circuit_board.jpeg', 'rb') as f:191 image_bytes = f.read()192193 system_instruction = "Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block)."194195 response = client.models.generate_content(196 model="gemini-robotics-er-2-preview",197 contents=[198 types.Part.from_bytes(199 data=image_bytes,200 mime_type='image/jpeg',201 ),202 """Read all visible component labels and markings on this circuit203 board. Reply in JSON: {"components": [{"label": val,204 "location": [y, x]}]}"""205 ],206 config = types.GenerateContentConfig(207 system_instruction=system_instruction,208 tools=[types.Tool(code_execution=types.ToolCodeExecution)],209 )210 )211212 print(response.text)213214215216## Image annotation217218The following example demonstrates how to use code execution to annotate an219image (e.g., drawing arrows for disposal instructions) and return the modified image.220221### Python222223 from google import genai224 from google.genai import types225226 client = genai.Client()227228 # Load your image229 with open('sorting.jpeg', 'rb') as f:230 image_bytes = f.read()231232 prompt = """233 Look at this image and return it as an annotated version using arrows of234 different colors to represent which items should go in which bins for235 disposal. You must return the final image to the API caller.236 """237238 response = client.models.generate_content(239 model="gemini-robotics-er-2-preview",240 contents=[241 types.Part.from_bytes(242 data=image_bytes,243 mime_type='image/jpeg',244 ),245 prompt246 ],247 config = types.GenerateContentConfig(248 tools=[types.Tool(code_execution=types.ToolCodeExecution)],249 )250 )251252 print(response.text)253254The following is an example image input.255256257258The model output would be similar to the following:259260 The annotated image shows the suggested disposal locations for the items on the table:261 - **Green bin (Compost/Organic)**: Green chili, red chili, grapes, and cherries.262 - **Blue bin (Recycling)**: Yellow crushed can and plastic container.263 - **Black bin (Trash)**: Chocolate bar wrapper, Welch's packet, and white tissue.264265## What's next266267- [Task orchestration](https://ai.google.dev/gemini-api/docs/robotics-orchestration) --- long-horizon tasks with custom robot APIs.268- [Robotics with streaming](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time bidirectional streaming (Gemini Robotics ER 2 only).269- [Video understanding](https://ai.google.dev/gemini-api/docs/robotics-video-progress) --- moment finding and progress classification (Gemini Robotics ER 2 only).