spb/doc-api

1<br />23Gemini Robotics ER models can plan tasks and reason about space inferring4which actions to take and which objects to move to complete a goal. This page5shows an example for [driving a pick-and-place](https://ai.google.dev/gemini-api/docs/calling-custom-robot-api)6operation through a custom robot API to orchestrate the task of placing an item7into a bowl. This example uses the standard Gemini ER 2 model, for a streaming8example, see the [Gemini ER 2 Streaming guide](https://ai.google.dev/gemini-api/docs/robotics-streaming).910For full runnable code, see the11[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).1213## Using a custom robot API1415This example demonstrates task orchestration with a custom robot API. It16introduces a mock API designed for a pick-and-place operation. The task is to17pick up a blue block and place it in an orange bowl:1819![An image of the block and bowl](https://ai.google.dev/static/gemini-api/docs/images/robotics/robot-api-example.png)2021This example uses the following mock robot API:2223### Python2425    def move(x, y, high):26      print(f"Mock Robot: Moving to coordinates: {x}, {y}, {'high above table' if high else 'down at table level'}")2728    def setGripperState(opened):29      print(f"Mock Robot: {'Opening gripper' if opened else 'Closing gripper'}")3031    robot_origin_y = 30032    robot_origin_x = 5003334    move_function = {35        "type": "function",36        "name": "move",37        "description": "Moves the arm to the given coordinates.",38        "parameters": {39            "type": "object",40            "properties": {41                "x": {"type": "integer", "description": "X coordinate relative to the origin"},42                "y": {"type": "integer", "description": "Y coordinate relative to the origin"},43                "high": {"type": "boolean", "description": "Set to True to lift the robot arm above the scene for avoiding obstacles. Set to False to place the gripper on the surface."}44            },45            "required": ["x", "y", "high"]46        }47    }4849    set_gripper_state_function = {50        "type": "function",51        "name": "setGripperState",52        "description": "Opens or closes the robot's gripper.",53        "parameters": {54            "type": "object",55            "properties": {56                "opened": {"type": "boolean", "description": "True opens the gripper, False closes the gripper."}57            },58            "required": ["opened"]59        }60    }6162The following example sends the prompt and image to the model with the tool63definitions. It then runs an agentic loop: after each model response, it64executes any requested function calls (`move`, `setGripperState`), returns the65results back to the model using `previous_interaction_id`, and repeats until the66model stops calling functions or the step limit is reached.6768### Python6970    prompt = (71        "You are a robotic arm with six degrees-of-freedom. "72        f"The origin point for calculating the moves is at normalized point y={robot_origin_y}, x={robot_origin_x}. "73        "Use this as the new (0,0) for calculating moves, allowing x and y to be negative.\n\n"74        "Find the blue block and the orange bowl. Calculate their coordinates relative to the origin.\n"75        "Perform a pick and place operation where you pick up the blue block and place it into the orange bowl. "76        "Call the appropriate sequence of functions to complete this operation."77    )7879    # 1. Initial Interaction80    interaction = client.interactions.create(81        model=MODEL_ID,82        input=[{"type": "user_input", "content": [83            {"type": "image", "data": img_b64, "mime_type": "image/png"},84            {"type": "text", "text": prompt}85        ]}],86        tools=[move_function, set_gripper_state_function],87        generation_config={"thinking_level": "low"}88    )8990    print("\n--- Executing Orchestrated Plan ---")9192    max_steps = 15 # Safety limit to prevent infinite loops93    step_count = 09495    # 2. The Agentic Loop96    while step_count < max_steps:97        step_count += 19899        # Check if the model wants to call any functions100        tool_calls = [step for step in interaction.steps if step.type == "function_call"]101102        if not tool_calls:103            # If no tools were called, the model is finished with the sequence104            print("Sequence complete.")105            if interaction.output_text:106                print(f"Model Summary: {interaction.output_text}")107            break108109        function_results = []110111        for step in tool_calls:112            function_name = step.name113            arguments = step.arguments114115            # Execute the mock function116            if function_name == "move":117                move(**arguments)118            elif function_name == "setGripperState":119                setGripperState(**arguments)120            else:121                print(f"Unknown function: {function_name}")122123            # 3. Create a result object to tell the model the function succeeded124            function_results.append({125                "type": "function_result",126                "name": step.name,127                "call_id": step.id,128                "result": [{"type": "text", "text": '{"status": "success"}'}]129            })130131        # 4. Send the results back to the model, passing previous_interaction_id132        # so it remembers the conversation history and generates the NEXT step133        interaction = client.interactions.create(134            model=MODEL_ID,135            previous_interaction_id=interaction.id,136            tools=[move_function, set_gripper_state_function],137            input=function_results138        )139140The following shows a possible output of the model based on the prompt and141the mock robot API. The output includes the output of the robot142function calls that the model sequenced together.143144    --- Executing Orchestrated Plan ---145    Mock Robot: Opening gripper146    Mock Robot: Moving to coordinates: 160, 440, high above table147    Mock Robot: Moving to coordinates: 160, 440, down at table level148    Mock Robot: Closing gripper149    Mock Robot: Moving to coordinates: 160, 440, high above table150    Mock Robot: Moving to coordinates: -250, 60, high above table151    Mock Robot: Moving to coordinates: -250, 60, down at table level152    Mock Robot: Opening gripper153    Mock Robot: Moving to coordinates: -250, 60, high above table154    Sequence complete.155    Model Summary: I have completed the task of picking up the blue block and placing it into the orange bowl.156157## What's next158159- [Robotics with streaming](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time streaming with function calling (Gemini Robotics ER 2 only).160- [Video understanding](https://ai.google.dev/gemini-api/docs/robotics-video-progress) --- track task progress from video (ER 2 only).161- [Spatial reasoning](https://ai.google.dev/gemini-api/docs/robotics-spatial) --- pointing, tracking, and bounding box examples.