Repository Public
1<br />23Gemini Robotics ER models can plan tasks and reason about space, inferring4which actions to take and which objects to move to complete a goal. This page5shows an example for [driving a pick-and-place](https://ai.google.dev/gemini-api/docs/generate-content/robotics-orchestration#calling-custom-robot-api)6operation through a custom robot API to orchestrate the task of placing an item7into a bowl.89For full runnable code, see the10[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).1112## Using a custom robot API1314This example demonstrates task orchestration with a custom robot API. It15introduces a mock API designed for a pick-and-place operation. The task is to16pick up a blue block and place it in an orange bowl:17181920This example uses the following mock robot API and tool definitions:2122### Python2324 from google import genai25 from google.genai import types2627 client = genai.Client()2829 def move(x, y, high):30 print(f"Mock Robot: Moving to coordinates: {x}, {y}, {'high above table' if high else 'down at table level'}")3132 def setGripperState(opened):33 print(f"Mock Robot: {'Opening gripper' if opened else 'Closing gripper'}")3435 robot_origin_y = 30036 robot_origin_x = 5003738 move_declaration = types.FunctionDeclaration(39 name="move",40 description="Moves the arm to the given coordinates.",41 parameters=types.Schema(42 type=types.Type.OBJECT,43 properties={44 "x": types.Schema(type=types.Type.INTEGER, description="X coordinate relative to the origin"),45 "y": types.Schema(type=types.Type.INTEGER, description="Y coordinate relative to the origin"),46 "high": types.Schema(type=types.Type.BOOLEAN, description="Set to True to lift the robot arm above the scene. Set to False to place the gripper on the surface."),47 },48 required=["x", "y", "high"],49 ),50 )5152 set_gripper_state_declaration = types.FunctionDeclaration(53 name="setGripperState",54 description="Opens or closes the robot's gripper.",55 parameters=types.Schema(56 type=types.Type.OBJECT,57 properties={58 "opened": types.Schema(type=types.Type.BOOLEAN, description="True opens the gripper, False closes the gripper."),59 },60 required=["opened"],61 ),62 )6364 robot_tools = types.Tool(function_declarations=[move_declaration, set_gripper_state_declaration])6566The following example sends the prompt and image to the model with the tool67definitions. It then runs an agentic loop: after each model response, it68executes any requested function calls (`move`, `setGripperState`), returns the69results back to the model, and repeats until the model stops calling functions70or the step limit is reached.7172### Python7374 with open("robot-api-example.png", "rb") as f:75 img_bytes = f.read()7677 prompt = (78 "You are a robotic arm with six degrees-of-freedom. "79 f"The origin point for calculating the moves is at normalized point y={robot_origin_y}, x={robot_origin_x}. "80 "Use this as the new (0,0) for calculating moves, allowing x and y to be negative.\n\n"81 "Find the blue block and the orange bowl. Calculate their coordinates relative to the origin.\n"82 "Perform a pick and place operation where you pick up the blue block and place it into the orange bowl. "83 "Call the appropriate sequence of functions to complete this operation."84 )8586 contents = [87 types.Content(role="user", parts=[88 types.Part.from_bytes(data=img_bytes, mime_type="image/png"),89 types.Part(text=prompt),90 ])91 ]9293 print("\n--- Executing Orchestrated Plan ---")9495 max_steps = 15 # Safety limit to prevent infinite loops96 step_count = 09798 # The Agentic Loop99 while step_count < max_steps:100 step_count += 1101102 response = client.models.generate_content(103 model="gemini-robotics-er-2-preview",104 contents=contents,105 config=types.GenerateContentConfig(106 tools=[robot_tools],107 thinking_config=types.ThinkingConfig(thinking_level="low"),108 ),109 )110111 # Add model response to conversation history112 contents.append(response.candidates[0].content)113114 # Check for function calls115 function_calls = [part for part in response.candidates[0].content.parts if part.function_call]116117 if not function_calls:118 # Model is done calling functions119 print("Sequence complete.")120 print(f"Model Summary: {response.text}")121 break122123 # Execute function calls and collect results124 function_response_parts = []125 for part in function_calls:126 fc = part.function_call127 if fc.name == "move":128 move(**fc.args)129 elif fc.name == "setGripperState":130 setGripperState(**fc.args)131132 function_response_parts.append(133 types.Part.from_function_response(134 name=fc.name,135 response={"status": "success"},136 )137 )138139 # Send function results back to model140 contents.append(types.Content(role="user", parts=function_response_parts))141142The following shows a possible output of the model based on the prompt and143the mock robot API. The output includes the output of the robot144function calls that the model sequenced together.145146 --- Executing Orchestrated Plan ---147 Mock Robot: Opening gripper148 Mock Robot: Moving to coordinates: 160, 440, high above table149 Mock Robot: Moving to coordinates: 160, 440, down at table level150 Mock Robot: Closing gripper151 Mock Robot: Moving to coordinates: 160, 440, high above table152 Mock Robot: Moving to coordinates: -250, 60, high above table153 Mock Robot: Moving to coordinates: -250, 60, down at table level154 Mock Robot: Opening gripper155 Mock Robot: Moving to coordinates: -250, 60, high above table156 Sequence complete.157 Model Summary: I have completed the task of picking up the blue block and placing it into the orange bowl.158159## What's next160161- [Robotics with streaming](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time streaming with function calling (Gemini Robotics ER 2 only).162- [Video understanding](https://ai.google.dev/gemini-api/docs/robotics-video-progress) --- track task progress from video (ER 2 only).163- [Spatial reasoning](https://ai.google.dev/gemini-api/docs/robotics-spatial) --- pointing, tracking, and bounding box examples.