spb/doc-api

1<br />23> [!NOTE]4> **Note:** Video understanding requires Gemini Robotics ER 2 (`gemini-robotics-er-2-preview`). It is not available on ER 1.6.56Gemini Robotics ER 2 can track task progress from continuous video feeds using7two capabilities:89- Moment finding: identifies the precise timestamp where a key event occurs.10- Progress classification: assigns each video to one of five completion brackets (0--20%, 20--40%, 40--60%, 60--80%, 80--100%).1112## Moment finding1314Moment finding identifies the exact video frame where a critical event occurs ---15for example, when a cup is full or a knot is tied. Robots use this to verify16success, sequence steps, and trigger corrections.1718The following example prompt asks the model to identify the completion moment19for a given task in a video:2021    from google import genai22    from google.genai import types2324    client = genai.Client()2526    with open("task_video.mp4", "rb") as f:27        video_bytes = f.read()2829    prompt = """30    At what timestamp (in seconds) does the task reach successful completion?31    Return a JSON object: {"completion_time_seconds": <float>}.32    If the task is not completed, return {"completion_time_seconds": null}.33    """3435    response = client.models.generate_content(36        model="gemini-robotics-er-2-preview",37        contents=[38            types.Part.from_bytes(data=video_bytes, mime_type="video/mp4"),39            prompt,40        ],41    )4243    print(response.text)4445The following shows example frames from a moment-finding video, with the46model identifying the task completion timestamp:4748![Example video frames showing the moment finding output with a timestamp overlay](https://ai.google.dev/static/gemini-api/docs/images/robotics/video-moment-finding.png)4950## Progress classification5152Progress classification assigns a video to one of five completion brackets:530--20%, 20--40%, 40--60%, 60--80%, or 80--100%. This gives robots real-time54situational awareness so they can adjust actions or retry failed steps without55restarting an entire workflow.5657The following example prompt asks the model to classify the current progress58level from a video:5960    from google import genai61    from google.genai import types6263    client = genai.Client()6465    with open("task_video.mp4", "rb") as f:66        video_bytes = f.read()6768    prompt = """69    Watch this video and classify the task progress level at the final frame.70    Return a JSON object with the progress bracket:71    {"progress_level": "0-20" | "20-40" | "40-60" | "60-80" | "80-100"}.72    """7374    response = client.models.generate_content(75        model="gemini-robotics-er-2-preview",76        contents=[77            types.Part.from_bytes(data=video_bytes, mime_type="video/mp4"),78            prompt,79        ],80    )8182    print(response.text)8384The following shows example frames from a progress classification video, with85the model assigning a progress bracket:8687![Example video frames showing the progress classification output with a progress bracket label](https://ai.google.dev/static/gemini-api/docs/images/robotics/video-progress-classification.png)8889## Examples9091For full runnable examples including multi-step task tracking, see the92[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).9394## What's next9596- [Live API for robotics](https://ai.google.dev/gemini-api/docs/robotics-streaming) --- real-time bidirectional streaming.97- [Task orchestration](https://ai.google.dev/gemini-api/docs/robotics-orchestration) --- long-horizon tasks with spatial reasoning.98- [Gemini Robotics ER overview](https://ai.google.dev/gemini-api/docs/robotics-overview) --- model comparison and capabilities.