computer_action

Drive a browser session with raw mouse and keyboard input at screen coordinates. Pass a single action for simple operations, or pass multiple actions to batch them into one request for lower latency.

Prefer execute_playwright_code for anything a selector can reach. It’s faster, deterministic, and doesn’t depend on the model’s ability to locate a target in an image. Reach for computer_action only when there’s no selector to target: canvas apps, embedded PDFs, native dialogs, and drag interactions.

Coordinates come from a screenshot call, and are only as accurate as the calling model’s pixel grounding — models vary widely here. Take a screenshot after acting to confirm the result.

Parameters

Parameter	Description
`session_id`	Browser session ID. Required.
`actions`	Ordered list of actions to perform. Required.

Action types

Type	Description
`click_mouse`	Click at `x`, `y`. Supports `button`, `click_type`, `num_clicks`, `hold_keys`.
`move_mouse`	Move the cursor to `x`, `y`.
`type_text`	Type `text`, with optional inter-key `delay`.
`press_key`	Press `keys` (X11 keysym names or combos like `Ctrl+t`, `Return`).
`scroll`	Scroll at `x`, `y` by `delta_x`/`delta_y` (positive = right/down).
`drag_mouse`	Drag along a `path` of `[x, y]` points.
`set_cursor`	Show or hide the cursor (`hidden`).
`sleep`	Wait `duration_ms` between steps when the page needs time to react.
`write_clipboard`	Write `text` to the clipboard.
`read_clipboard`	Return the current clipboard contents.
`get_mouse_position`	Return the current cursor position.

read_clipboard and get_mouse_position return data, so they must come last. To capture the page, call the separate screenshot tool.

Example

{
  "session_id": "browser_abc123",
  "actions": [
    { "type": "click_mouse", "click_mouse": { "x": 420, "y": 300 } },
    { "type": "type_text", "type_text": { "text": "kernel browsers" } },
    { "type": "press_key", "press_key": { "keys": ["Return"] } },
    { "type": "sleep", "sleep": { "duration_ms": 1000 } }
  ]
}

Then call screenshot to see the result.

Getting Started

Connect Your Client

Tools

Resources & Prompts

Reference

computer_action

Parameters

Action types

Example

​Parameters

​Action types

​Example

Parameters

Action types

Example