Repository Public
1# Computer use23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Computer use lets a model operate browser and desktop interfaces. Use it to fill out forms, test user flows, or complete tasks in applications through their UI.67You provide the environment and execute the model's requests. The model uses screenshots and other tool results to decide what to do next. Choose how to connect it to your application:89<a id="choose-an-integration-path"></a>1011- **Code execution:** The model writes code that uses a library such as PyAutoGUI or Playwright to operate the interface. One call can combine actions, loops, or conditional logic.12- **The computer tool:** The model returns structured mouse and keyboard actions that your application translates into browser or desktop input.1314For [GPT-6 Astra](https://developers.openai.com/api/docs/models/gpt-6-astra), we recommend code execution. The `computer` tool remains supported as an alternative.1516<a id="option-2-use-a-custom-tool-or-harness"></a>17<a id="use-your-own-ui-tools"></a>18<a id="use-an-existing-tool-interface"></a>1920If you already expose UI operations through [function calling](https://developers.openai.com/api/docs/guides/function-calling) or [remote MCP tools](https://developers.openai.com/api/docs/guides/tools-connectors-mcp), you can keep that interface. See [Use your own UI tools](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#use-your-own-ui-tools) for the differences in how those integrations execute tools and return results.2122<a id="expose-a-code-execution-tool"></a>23<a id="option-3-use-a-code-execution-harness"></a>24<a id="use-a-code-execution-harness"></a>2526## Use code execution2728A code-execution integration gives the model a function tool that accepts a script. Your application runs the script in an isolated browser or desktop environment and returns its output, including screenshots. Keep the environment available between calls so the model can build on earlier work.2930<a id="before-running-the-examples"></a>3132### Run the sample app3334The [CUA sample app](https://github.com/openai/openai-cua-sample-app#first-run) includes JavaScript/Playwright and Python/PyAutoGUI implementations, with local tasks and a shared console:35361. Follow the setup instructions for your chosen implementation in an isolated environment.372. Choose a built-in scenario and start a run. 383. Inspect the actions, screenshots, and final state to assess whether the task succeeded.3940Use the app's README for installation, desktop permissions, and supported environments. Review [Run safely](#run-safely) before adapting it to real sites or accounts.4142<a id="code-execution-harness-examples"></a>4344### Connect your own runtime4546The following example shows the API loop for a runtime you provide. Python and Ruby send Python code to a desktop runtime that uses PyAutoGUI; JavaScript uses Playwright to operate a browser. Each client exposes an ordinary function tool and returns text or images with the original `call_id`.4748The `execute_in_sandbox` or `executeInSandbox` helper sends code to your execution environment and returns its observations. It must preserve the browser or desktop session, enforce execution limits, and apply your permission rules. These are integration examples, separate from running the sample app.49505152Python5354 Run computer use with code execution5556```python57import json58import uuid5960from openai import OpenAI61from openai.types.responses import (62 FunctionToolParam,63 ResponseInputParam,64)6566def run_computer_use(endpoint, prompt, model="gpt-6-astra"):67 client = OpenAI()68 session_id = str(uuid.uuid4())69 tools: list[FunctionToolParam] = [70 {71 "type": "function",72 "name": "exec_py",73 "description": (74 "Run Python in a persistent desktop. Variables persist across calls. "75 "PyAutoGUI operations are synchronous. Available: pyautogui, time, "76 "log(value), and display(PIL_image). Inspect the screen with "77 "display(pyautogui.screenshot()) before acting. Use screenshot "78 "coordinates and check the screen after a short group of actions. "79 "Keep screenshots in memory and PyAutoGUI's fail-safe enabled."80 ),81 "parameters": {82 "type": "object",83 "properties": {"code": {"type": "string"}},84 "required": ["code"],85 "additionalProperties": False,86 },87 "strict": True,88 }89 ]90 next_input: ResponseInputParam = [{"role": "user", "content": prompt}]91 previous_response_id = None9293 for turn in range(20):94 response = client.responses.create(95 model=model,96 tools=tools,97 input=next_input,98 previous_response_id=previous_response_id,99 )100 if response.status != "completed":101 raise RuntimeError(f"Response stopped with status: {response.status}")102103 calls = [item for item in response.output if item.type == "function_call"]104 if not calls and any(105 item.type == "message" and item.phase != "commentary"106 for item in response.output107 ):108 print(response.output_text)109 return110 if turn == 19:111 raise RuntimeError(112 "The task reached the 20-response limit. Inspect the last result."113 )114115 next_input = []116 for call in calls:117 if call.name != "exec_py":118 raise ValueError(f"Unexpected tool: {call.name}")119 code = json.loads(call.arguments)["code"]120 output = execute_in_sandbox(code, session_id, endpoint)121 next_input.append(122 {123 "type": "function_call_output",124 "call_id": call.call_id,125 "output": output,126 }127 )128 previous_response_id = response.id129```130131 132133 134135 136JavaScript137138 Run computer use with code execution139140```javascript141import { randomUUID } from "node:crypto";142import OpenAI from "openai";143144async function runComputerUse(endpoint, prompt, model = "gpt-6-astra") {145 const client = new OpenAI();146 const sessionId = randomUUID();147148 const tools = [149 {150 type: "function",151 name: "exec_js",152 description: `Run JavaScript in a persistent browser. Available: Playwright's153browser, context, and page objects; console.log(value); and display(base64Image).154Save reusable variables on globalThis. Inspect a screenshot before acting and155check the screen after a short group of actions. Keep screenshots in memory.156Use top-level await for async operations. Return images with display() and concise157text with console.log(). The context viewport is 1440x900.`,158 parameters: {159 type: "object",160 properties: { code: { type: "string" } },161 required: ["code"],162 additionalProperties: false,163 },164 strict: true,165 },166 ];167168 let nextInput = [{ role: "user", content: prompt }];169 let previousResponseId;170171 for (let turn = 0; turn < 20; turn++) {172 const response = await client.responses.create({173 model,174 tools,175 input: nextInput,176 previous_response_id: previousResponseId,177 reasoning: { effort: "low" },178 });179 if (response.status !== "completed") {180 throw new Error(`Response stopped with status: ${response.status}`);181 }182 const calls = response.output.filter(183 (item) => item.type === "function_call"184 );185 if (186 calls.length === 0 &&187 response.output.some(188 (item) => item.type === "message" && item.phase !== "commentary"189 )190 ) {191 console.log(response.output_text);192 return;193 }194 if (turn === 19) {195 throw new Error(196 "The task reached the 20-response limit. Inspect the last result."197 );198 }199200 nextInput = [];201 for (const call of calls) {202 if (call.name !== "exec_js")203 throw new Error(`Unexpected tool: ${call.name}`);204 const { code } = JSON.parse(call.arguments);205 const output = await executeInSandbox(code, sessionId, endpoint);206 nextInput.push({207 type: "function_call_output",208 call_id: call.call_id,209 output,210 });211 }212 previousResponseId = response.id;213 }214}215```216217 218219 220221 222Ruby223224 Run computer use with code execution225226```ruby227require "json"228require "openai"229require "securerandom"230231def run_computer_use(endpoint, prompt)232 client = OpenAI::Client.new233 session_id = SecureRandom.uuid234 tools = [235 {236 type: :function,237 name: "exec_py",238 description: "Run Python in a persistent desktop. Variables persist across calls. PyAutoGUI operations are synchronous. Available: pyautogui, time, log(value), and display(PIL_image). Inspect the screen with display(pyautogui.screenshot()) before acting. Use screenshot coordinates and check the screen after a short group of actions. Keep screenshots in memory and PyAutoGUI's fail-safe enabled.",239 parameters: {240 type: :object,241 properties: { code: { type: :string } },242 required: ["code"],243 additionalProperties: false244 },245 strict: true246 }247 ]248 next_input = []249 next_input << {250 role: :user,251 content: prompt252 }253 history = {}254 20.times do |turn|255 response = client.responses.create(256 model: "gpt-6-astra", tools: tools, input: next_input, previous_response_id: history[:id]257 )258 raise "Response stopped with status: #{response.status}" unless response.status == OpenAI::Responses::ResponseStatus::COMPLETED259260 calls = response.output.grep(OpenAI::Responses::ResponseFunctionToolCall)261 if calls.empty? && response.output.any? { |item| item.is_a?(OpenAI::Responses::ResponseOutputMessage) && item.phase != :commentary }262 puts(response.output_text)263 return response264 end265 raise "The task reached the 20-response limit" if turn == 19266267 next_input.clear268 calls.each do |call|269 raise "Unexpected tool: #{call.name}" unless call.name == "exec_py"270271 code = JSON.parse(call.arguments).fetch("code")272 raise "Expected Python source text" unless code.is_a?(String)273274 output = execute_in_sandbox(code, session_id, endpoint)275 next_input << {276 type: :function_call_output,277 call_id: call.call_id,278 output: output279 }280 end281 history[:id] = response.id282 end283end284```285286287288<a id="connect-to-your-execution-service"></a>289290For a complete client adapter and the expected text and image output shape, see [Connect to your execution service](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#connect-to-your-execution-service). The service interface in those examples belongs to your application; it is not an OpenAI-hosted endpoint.291292### Preserve state and return observations293294Keep the browser or desktop session alive between calls. A persistent Python or JavaScript namespace can also preserve variables. Describe the available objects and helpers in the tool definition so the model knows what it can use.295296Give the model a current screenshot when the UI state is unknown. After a short group of actions, return another screenshot so it can check the result. Keep images in memory and use `detail: "original"` to preserve resolution. If you downscale a screenshot, map the model's coordinates back to the environment's coordinate space before executing actions. See [Screenshot capture and resolution](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#capture-screenshots).297298The API conversation and the execution environment have separate state. Preserve tool calls and their outputs in the conversation, and keep the corresponding environment available in your application. Continuing a response does not restore a browser session, login state, or runtime variables.299300<a id="provide-the-environment-and-control-the-loop"></a>301<a id="option-1-run-the-built-in-computer-use-loop"></a>302303## Use the computer tool304305Use this alternative when your integration expects structured actions instead of generated code. For the recommended approach, start with [code execution](#use-code-execution).306307To try this path, follow the [same sample-app setup](https://github.com/openai/openai-cua-sample-app#first-run), select **Native** mode, and run a built-in scenario. Use a model that supports the [computer tool](https://developers.openai.com/api/docs/models).308309The API exchange has three steps: send a task, execute the returned actions, and return a screenshot. The snippets here use a page with a **Show filters** control and a search field. Adapt that task to your own interface when integrating the tool.310311<a id="prepare-a-safe-environment"></a>312<a id="1-prepare-your-browser-or-desktop"></a>313<a id="create-a-docker-image"></a>314315For environment setup and action handlers, use the [integration recipes](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#prepare-an-environment).316317<a id="1-send-the-first-request"></a>318<a id="2-send-the-task"></a>319<a id="1-send-the-task"></a>320321### Send the task322323Enable `computer` in the `tools` array and describe the result you want:324325Send a computer request326327```javascript328import OpenAI from "openai";329330const client = new OpenAI();331332const response = await client.responses.create({333 model: "gpt-5.6-sol",334 tools: [{ type: "computer" }],335 input:336 "Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.",337});338339console.log(JSON.stringify(response.output, null, 2));340```341342```python343from openai import OpenAI344345client = OpenAI()346347response = client.responses.create(348 model="gpt-5.6-sol",349 tools=[{"type": "computer"}],350 input="Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.",351)352353print(response.output)354```355356```go357package main358359import (360 "context"361 "fmt"362363 "github.com/openai/openai-go/v3"364 "github.com/openai/openai-go/v3/responses"365)366367func main() {368 client := openai.NewClient()369 response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{370 Model: "gpt-5.6-sol",371 Tools: []responses.ToolUnionParam{{OfComputer: &responses.ComputerToolParam{}}},372 Input: responses.ResponseNewParamsInputUnion{OfString: openai.String("Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.")},373 })374 if err != nil {375 panic(err)376 }377 fmt.Println(response.Output)378}379```380381```java382import com.openai.client.OpenAIClient;383import com.openai.client.okhttp.OpenAIOkHttpClient;384import com.openai.core.JsonValue;385import com.openai.models.responses.ResponseCreateParams;386import java.util.List;387import java.util.Map;388389ResponseCreateParams params =390 ResponseCreateParams.builder()391 .model("gpt-5.6-sol")392 .input(393 "Open the Filters panel if needed, then search for penguin. Use the computer tool for UI interaction.")394 .putAdditionalBodyProperty("tools", JsonValue.from(List.of(Map.of("type", "computer"))))395 .build();396397client.responses().create(params).output().forEach(System.out::println);398```399400```ruby401require "openai"402403client = OpenAI::Client.new404response = client.responses.create(405 model: "gpt-5.6-sol",406 input: "Open the Filters panel if needed, then search for penguin. Use the computer tool for UI interaction.",407 tools: [{ type: :computer }]408)409410puts(response.output)411```412413414<a id="2-handle-screenshot-first-turns"></a>415<a id="3-inspect-the-requested-actions"></a>416<a id="3-run-every-returned-action"></a>417<a id="2-execute-the-requested-actions"></a>418419### Execute the requested actions420421A `computer_call` contains an ordered `actions` array. For example, this call selects the search field and types `penguin`:422423Batched actions in one turn424425```json426{427 "output": [428 {429 "type": "computer_call",430 "call_id": "call_002",431 "actions": [432 { "type": "click", "button": "left", "x": 405, "y": 157 },433 { "type": "type", "text": "penguin" }434 ],435 "status": "completed"436 }437 ]438}439```440441442Your action handler translates these requests into browser or operating system input. Execute permitted actions in order, then capture the updated screen. The model can request `click`, `double_click`, `drag`, `move`, `scroll`, `keypress`, `type`, `wait`, or `screenshot`.443444The first call may contain only a `screenshot` action. In that case, capture the current screen and return it without changing the UI. A call's `status: "completed"` means the model has finished generating that call; your application still needs to execute it.445446<a id="possible-computer-use-actions"></a>447<a id="supported-actions"></a>448<a id="implement-action-handlers"></a>449450Use the [action-handler examples](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#implement-action-handlers) for key mappings, drag paths, and modifier keys.451452<a id="4-capture-and-return-the-updated-screenshot"></a>453<a id="4-return-the-updated-screen"></a>454<a id="3-return-the-screenshot"></a>455456### Return the screenshot457458Return a `computer_call_output` whose `call_id` matches the call you handled. Use `previous_response_id` to continue the model conversation:459460Send the updated screenshot461462```javascript463import OpenAI from "openai";464465const client = new OpenAI();466467async function sendComputerScreenshot(response, callId, screenshotBase64) {468 const output = {469 type: "computer_screenshot",470 image_url: `data:image/png;base64,${screenshotBase64}`,471 detail: "original",472 };473474 return await client.responses.create({475 model: "gpt-5.6-sol",476 tools: [{ type: "computer" }],477 previous_response_id: response.id,478 input: [479 {480 type: "computer_call_output",481 call_id: callId,482 output,483 },484 ],485 });486}487```488489```python490from openai import OpenAI491492client = OpenAI()493494495def send_computer_screenshot(response, call_id, screenshot_base64):496 return client.responses.create(497 model="gpt-5.6-sol",498 tools=[{"type": "computer"}],499 previous_response_id=response.id,500 input=[501 {502 "type": "computer_call_output",503 "call_id": call_id,504 "output": {505 "type": "computer_screenshot",506 "image_url": f"data:image/png;base64,{screenshot_base64}",507 "detail": "original",508 },509 }510 ],511 )512```513514```go515package main516517import (518 "context"519 "fmt"520521 "github.com/openai/openai-go/v3"522 "github.com/openai/openai-go/v3/responses"523)524525func main() {526 client := openai.NewClient()527 response, err := sendComputerScreenshot(client, "resp_abc123", "call_abc123", "<base64 bytes here>")528 if err != nil {529 panic(err)530 }531 fmt.Println(response.Output)532}533534func sendComputerScreenshot(client openai.Client, responseID string, callID string, screenshotBase64 string) (*responses.Response, error) {535 screenshot := responses.ResponseComputerToolCallOutputScreenshotParam{536 ImageURL: openai.String("data:image/png;base64," + screenshotBase64),537 }538 screenshot.SetExtraFields(map[string]any{"detail": "original"})539 return client.Responses.New(context.Background(), responses.ResponseNewParams{540 Model: "gpt-5.6-sol",541 Tools: []responses.ToolUnionParam{{OfComputer: &responses.ComputerToolParam{}}},542 PreviousResponseID: openai.String(responseID),543 Input: responses.ResponseNewParamsInputUnion{OfInputItemList: responses.ResponseInputParam{544 responses.ResponseInputItemParamOfComputerCallOutput(callID, screenshot),545 }},546 })547}548```549550```java551import com.openai.client.OpenAIClient;552import com.openai.client.okhttp.OpenAIOkHttpClient;553import com.openai.core.JsonValue;554import com.openai.models.responses.ResponseComputerToolCallOutputScreenshot;555import com.openai.models.responses.ResponseCreateParams;556import com.openai.models.responses.ResponseInputItem;557import java.util.List;558import java.util.Map;559560String responseId = "resp_abc123";561562String computerCallId = "call_abc123";563564String screenshotBase64 = "<base64 bytes here>";565566ResponseCreateParams params =567 ResponseCreateParams.builder()568 .model("gpt-5.6-sol")569 .input(570 ResponseCreateParams.Input.ofResponse(571 List.of(572 ResponseInputItem.ofComputerCallOutput(573 ResponseInputItem.ComputerCallOutput.builder()574 .callId(computerCallId)575 .output(576 ResponseComputerToolCallOutputScreenshot.builder()577 .imageUrl("data:image/png;base64," + screenshotBase64)578 .putAdditionalProperty("detail", JsonValue.from("original"))579 .build())580 .build()))))581 .previousResponseId(responseId)582 .putAdditionalBodyProperty("tools", JsonValue.from(List.of(Map.of("type", "computer"))))583 .build();584585client.responses().create(params).output().forEach(System.out::println);586```587588```ruby589require "openai"590591client = OpenAI::Client.new592response = client.responses.create(593 model: "gpt-5.6-sol",594 previous_response_id: "resp_abc123",595 input: [596 {597 type: :computer_call_output,598 call_id: "call_abc123",599 output: {600 type: :computer_screenshot,601 image_url: "data:image/png;base64,<base64 bytes here>",602 detail: :original603 }604 }605 ],606 tools: [{ type: :computer }]607)608609puts(response.output)610```611612613The same [screenshot and state guidance](#preserve-state-and-return-observations) applies to this loop. Keep the environment available while `previous_response_id` continues the model conversation.614615<a id="5-repeat-until-the-tool-stops-calling"></a>616<a id="5-continue-and-verify-the-result"></a>617618Continue until the model stops returning `computer_call` items. Inspect the remaining output for an answer, a request for help, or another tool call, and verify the result in the application. For this example, the Filters panel should be open and the search field should contain `penguin`.619620See [Repeat the computer-use loop](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#repeat-the-computer-use-loop) for the loop skeleton, including its required action and screenshot helpers.621622<a id="handle-user-confirmation-and-consent"></a>623<a id="keep-a-human-in-the-loop"></a>624<a id="restrict-the-environment"></a>625626## Run safely627628Computer use can affect real accounts and data. Apply these controls in your application and execution environment as well as in the model's instructions:629630- **Restrict the environment.** Use an isolated browser or VM and an allow list of sites and actions. Keep access limited to what the task needs.631- **Treat screen content as untrusted.** Text in a page, document, or tool result cannot grant permission or override the user's instructions.632- **Confirm consequential actions.** Keep users in control of purchases, data transmission, destructive changes, and other actions that are hard to reverse. Typing sensitive information into a form counts as transmission.633- **Bound and verify the run.** Set step, time, or cost limits, support cancellation, and check the actual outcome instead of relying only on the model's final answer.634635<a id="treat-only-direct-user-instructions-as-permission"></a>636<a id="confirm-at-the-point-of-risk"></a>637<a id="use-the-right-confirmation-level"></a>638<a id="hand-off-required"></a>639<a id="always-confirm-at-action-time"></a>640<a id="pre-approval-can-be-enough"></a>641<a id="protect-sensitive-data"></a>642<a id="prompt-patterns-you-can-add-to-your-agent-instructions"></a>643<a id="distinguish-direct-user-intent-from-untrusted-third-party-content"></a>644<a id="delay-confirmation-until-the-exact-risky-action"></a>645<a id="require-explicit-consent-before-transmitting-sensitive-data"></a>646<a id="stop-and-escalate-when-the-model-sees-prompt-injection-or-suspicious-instructions"></a>647648See the [confirmation and consent guidance](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#handle-user-confirmation-and-consent) for specific approval requirements, human handoff, and prompt examples.649650<a id="migration-from-computer-use-preview"></a>651<a id="explore-more-examples"></a>652653## Next steps654655- Use the [integration recipes](https://developers.openai.com/api/docs/guides/tools-computer-use-integration) for environment setup, action handlers, screenshot capture, and execution-service adapters.656- Follow [Migration from computer-use-preview](https://developers.openai.com/api/docs/guides/tools-computer-use-integration#migration-from-computer-use-preview) when updating an older integration.657- Explore the [CUA sample app](https://github.com/openai/openai-cua-sample-app) for complete browser and desktop workflows.