# Gemini — Computer Use (tools[].computerUse, Preview)

Status: DOCUMENTED + ACCOUNT_RESTRICTED (2026-09-18: our key got HTTP 429 quota errors — see "Live verification" section at the end)

Sources:

• https://ai.google.dev/gemini-api/docs/generate-content/computer-use · https://ai.google.dev/gemini-api/docs/computer-use
• https://ai.google.dev/api/generate-content (#ComputerUse #Environment #SafetyPolicy)
• https://ai.google.dev/gemini-api/docs/models/gemini-2.5-computer-use-preview-10-2025 · https://ai.google.dev/gemini-api/docs/pricing · https://ai.google.dev/gemini-api/docs/changelog
• Reference implementation: https://github.com/google/computer-use-preview

Last verified: 2026-09-18 (docs only)

# 1. Models

# 2. Tool config (tools[].computerUse)

Extra: add tools[].functionDeclarations for custom actions (e.g. yield_to_user(reason)); tune generationConfig.thinkingConfig.thinkingLevel (lower = faster). No display size needed — coordinates are normalized 0–999 and you scale them to your viewport.

# 3. Predefined actions (functionCall names)

# Browser (ENVIRONMENT_BROWSER, Gemini 3.x)

# Mobile (ENVIRONMENT_MOBILE): open_app(app_name), click, list_apps, wait, go_back, type, drag_and_drop, long_press(x,y,seconds=2), press_key, take_screenshot.

# Desktop (ENVIRONMENT_DESKTOP): browser set minus go_back/navigate/go_forward.

# Legacy (gemini-2.5-computer-use-preview-10-2025)

# 4. Safety decisions

Model: functionCall.args.safety_decision = {"decision": "require_confirmation", "explanation": "..."} (regular/allowed, require_confirmation, or blocked). Client: prompt the user; if confirmed, include "safety_acknowledgement": true in the functionResponse response object; otherwise terminate. Always implement this handling regardless of disabledSafetyPolicies.

# 5. Loop (pseudo-code, Playwright)

from playwright.sync_api import sync_playwright
from google import genai; from google.genai import types
client = genai.Client(); W, H = 1440, 900
cfg = types.GenerateContentConfig(tools=[types.Tool(computer_use=types.ComputerUse(environment="ENVIRONMENT_BROWSER"))])
with sync_playwright() as p:
    page = p.chromium.launch().new_context(viewport={"width": W, "height": H}).new_page(); page.goto("https://www.google.com")
    history = [types.Content(role="user", parts=[types.Part(text="Search for 'Gemini API'."),
               types.Part.from_bytes(data=page.screenshot(type="png"), mime_type="image/png")])]
    for _ in range(10):
        r = client.models.generate_content(model="gemini-3.8-flash", contents=history, config=cfg)
        history.append(r.candidates[0].content)                     # keeps thoughtSignature + id
        calls = [pt.function_call for pt in r.candidates[0].content.parts if pt.function_call]
        if not calls: break
        responses = []
        for fc in calls:
            a = dict(fc.args); result = {}
            if "safety_decision" in a: result["safety_acknowledgement"] = True   # after asking the user!
            if fc.name in ("click", "click_at"): page.mouse.click(a["x"] / 1000 * W, a["y"] / 1000 * H)
            elif fc.name in ("type", "type_text_at"):
                if "x" in a: page.mouse.click(a["x"] / 1000 * W, a["y"] / 1000 * H)
                page.keyboard.type(a["text"]);  a.get("press_enter") and page.keyboard.press("Enter")
            elif fc.name == "navigate": page.goto(a["url"])
            elif fc.name == "scroll": page.mouse.wheel(0, a.get("magnitude_in_pixels", 300) * (1 if a["direction"] == "down" else -1))
            page.wait_for_load_state("load", timeout=5000)
            responses.append(types.Part(function_response=types.FunctionResponse(name=fc.name, id=fc.id,
                response={"url": page.url, **result},
                parts=[types.FunctionResponsePart(inline_data=types.FunctionResponseBlob(mime_type="image/png", data=page.screenshot(type="png")))])))
        history.append(types.Content(role="user", parts=responses))

The official guide's get_function_responses sample returns the screenshot as an image content block alongside {"url": current_url, ...}; the generateContent shape is functionResponse.parts[].inlineData (Gemini 3 multimodal function responses).

# 6. Pricing / limits / security

• Pricing: regular model token pricing (screenshots = image tokens); free tier not available; legacy 2.5 CU model has its own table.
• Preview: "may contain errors and security vulnerabilities" — avoid critical decisions, sensitive data, irreversible actions.
• Run in a sandboxed VM/container (reference Docker sandbox); human-in-the-loop on require_confirmation; custom safety system instructions; enable prompt-injection detection where available; not in the Live API.

# 7. Minimal request

curl -s "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash-lite:generateContent" \
  -H "x-goog-api-key: $GEMINI_API_KEY" -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"Search for Gemini API on Google."}]}],"tools":[{"computerUse":{"environment":"ENVIRONMENT_BROWSER"}}]}'

const r = await ai.models.generateContent({ model: 'gemini-3.5-flash-lite', contents: "Search for 'Gemini API' on Google.",
  config: { tools: [{ computerUse: { environment: 'ENVIRONMENT_BROWSER', excludedPredefinedFunctions: ['drag_and_drop'] } }] } });
console.log(r.functionCalls);

# Live verification (2026-09-18)

• Legacy generateContent tool tools:[{"computerUse":{"environment":"ENVIRONMENT_BROWSER"}}] on gemini-2.5-computer-use-preview-10-2025 with a 1×1 PNG → HTTP 429 Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_input_token_count, limit: 0, model: computer-use-preview → ACCOUNT_RESTRICTED (tmp-live/gemini-tools/f_computer_use_generateContent.json).
• Interactions API tools:[{"type":"computer_use","environment":"browser"}] on gemini-3.5-flash-lite, input "Do nothing. Reply DONE." → HTTP 200, status:"completed", steps [thought, model_output "DONE"] — the tool declaration is accepted on a non-computer-use model; no action was requested so the function_call shape was not observed live (f2_computer_use_interactions.json).
• The client loop (Playwright, function_result with screenshot + url, safety_decision acknowledgement) in examples/gemini/tools/computer-use/computer_use_interactions.py --loop is DOCUMENTED, not executed.