Drive the desktop: click, right_click, double_click, move, drag, scroll, type, press, or hotkey. Address by pixel coordinates read off the screenshot (top-left origin).
actionstring*required
What to do: `click`, `right_click`, `double_click`, `move`, `drag`, `scroll`, `type`, `press`, or `hotkey`.
buttonstring
Mouse button for `drag` (`left`, `right`, `middle`). Default `left`.
keysstring[]
Keys for `press`/`hotkey`: a single key (`["enter"]`) or a chord (`["ctrl", "c"]`).
scroll_xinteger· int64
Horizontal scroll amount for `action=scroll`. Default 0.
scroll_yinteger· int64
Vertical scroll amount for `action=scroll`; positive scrolls down. Default 3.
session_idstring*required
textstring
Text to type, for `action=type`.
to_xinteger· int64
Drag end x (with `to_y`), for `action=drag`.
to_yinteger· int64
Drag end y.
xinteger· int64
Target x (pixels). Required for click/right_click/double_click/move and the drag start; the scroll anchor defaults to screen center.
yinteger· int64
Target y (pixels).
detailstring*required
What was done, for the transcript.
computer::displays
functionList the local displays (index, name, primary, size). Pass the chosen index as `monitor` to computer::sessions::start to drive that display; omit it to use the display under the cursor. Native host only.
displaysobject[]*required
heightinteger· uint32*requiredmin 0
Logical height in points.
indexinteger· uint32*requiredmin 0
Index to pass as `monitor` to `computer::sessions::start`.
widthinteger· uint32*requiredmin 0
Logical width in points.
Internal: newest screencast frame, or nothing when since_frame is still current. No capture round-trip; poll fast. Not an agent function.
session_idstring*required
since_frameinteger· uint64min 0
Frame cursor from the previous read; when the newest frame still has this seq the response omits `frame` (nothing changed).
activeboolean*required
False when no screencast is running (call screencast::start first).
framestring
Base64 image of the newest frame; absent when `since_frame` is still current or no frame has arrived yet.
frame_seqinteger· uint64*requiredmin 0
heightinteger· uint32*requiredmin 0
mimestring*required
Image mime of the frame bytes.
timestampinteger· int64*required
widthinteger· uint32*requiredmin 0
computer::observe
functionCapture the desktop plus, optionally, the accessibility tree. Use include_a11y on macOS guests for a machine-readable element tree; elsewhere prefer computer::screenshot.
include_a11yboolean
Also fetch the accessibility tree. macOS returns a real tree; other guests may return a stub or nothing (the field is then omitted).
session_idstring*required
accessibilityunknown
Accessibility tree, present only when requested and the guest exposes one.
detailsobject*required
screenobject*required
Desktop pixel dimensions. Coordinates handed to pointer actions are in this space: integer pixels, top-left origin, 1:1 with the screenshot.
heightinteger· uint32*requiredmin 0
widthinteger· uint32*requiredmin 0
session_idstring*required
Internal: reload computer settings from the authoritative configuration on change.
Internal: start pushing live desktop frames onto the computer:frames stream for the console viewport. Console-UI plumbing; agents use computer::screenshot. Not an agent function.
session_idstring*required
Internal: stop the live frame push. Idempotent. Not an agent function.
session_idstring*required
computer::screenshot
functionCapture the desktop as a viewable image. This is how you see the screen before acting; the coordinate space of computer::act is this image's pixels (top-left origin).
session_idstring*required
detailsobject*required
heightinteger· uint32*requiredmin 0
mimestring*required
Detected image mime (`image/png` or `image/jpeg`).
session_idstring*required
widthinteger· uint32*requiredmin 0
computer::sessions::list
functionList live computer sessions with their endpoint, guest OS, and screen size.
sessionsobject[]*required
created_msinteger· int64*required
last_used_msinteger· int64*required
screenobject*required
Desktop pixel dimensions. Coordinates handed to pointer actions are in this space: integer pixels, top-left origin, 1:1 with the screenshot.
heightinteger· uint32*requiredmin 0
widthinteger· uint32*requiredmin 0
screencast_activeboolean*required
True while a live screen stream is running for this session.
session_idstring*required
Start a computer-use session and return its session_id. Pass `image` to boot a fresh desktop in an iii-sandbox microVM (fixed virtual display, no host setup), an `endpoint` to drive a desktop through its guest executor, or omit both to drive the local machine. Sessions are durable; stop them with computer::sessions::stop when done.
endpointstring
Desktop to drive when not using a sandbox `image`. Omit (and leave `image` unset) to drive the local machine this worker runs on (native driver, nothing else to run). Pass the endpoint of a desktop guest's executor (a `ws`/`wss`/`http`/`https` url or a bare `host:port`) to drive a remote desktop; falls back to the configured `default_endpoint` when omitted.
imagestring
Boot a fresh desktop inside an iii-sandbox microVM from this OCI image (a sandbox preset name or `custom_images` key) and drive it through iii primitives alone. A fixed virtual display means 1:1 coordinates, no HiDPI or multi-monitor ambiguity. Falls back to the configured `sandbox_image` when omitted. Takes precedence over `endpoint`.
monitorinteger· uint32min 0
Display index (from `computer::displays`) for a native session. Omit to drive the display under the cursor. Ignored for a remote `endpoint` or a sandbox `image`.
osstring
Guest OS label recorded on the session and surfaced in `session-started` (`linux`, `macos`, `windows`, `android`). Omit and each driver labels itself: a sandbox session is `linux`, a remote one takes the configured `os`, and a native one takes this host's OS.
endpointstring*required
What the session drives: `native` for the local machine, or the normalized remote endpoint.
screenall of*required
Desktop pixel dimensions; the coordinate space for `computer::act`.
all of (1)
variant 1
heightinteger· uint32*requiredmin 0
widthinteger· uint32*requiredmin 0
session_idstring*required
Pass this to every other computer function.
computer::sessions::stop
functionStop a computer session and close its driver connection. Idempotent: stopping an unknown or already-stopped session succeeds with was_running=false.
session_idstring*required
Session to stop. Stopping an unknown or already-stopped id succeeds.
was_runningboolean*required
False when the session was already gone.
computer::ui-content
functionServe the computer worker's injected console UI assets (content function for its console:script / console:style triggers).
pathstring*required
The asset path from the trigger config (e.g. `state/page.js`).
contentstring*required
The asset source, verbatim.
content_typestring*required
MIME type the console should serve the asset with.
