Skip to content
Registry

computer

v0.1.0

Drive a full desktop on the iii bus. Start a session on the local machine, a sandboxed desktop, or a remote one, screenshot it, click and type by coordinate, read app accessibility trees and act on elements by reference (native sessions), and stream the live screen.

automationcomputer-usedesktopguiscreenshot

Install

bash
iii trigger compose::add worker=computer@0.1.0

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions12

computer::act

Function

Drive the desktop: click, right_click, double_click, move, drag, scroll, type, press, or hotkey. Address by pixel coordinates read off the screenshot (top-left origin).

Request
  • actionstringrequired

    What to do: `click`, `right_click`, `double_click`, `move`, `drag`, `scroll`, `type`, `press`, or `hotkey`.

  • buttonstring

    Mouse button for `drag` (`left`, `right`, `middle`). Default `left`.

  • keysstring[]

    Keys for `press`/`hotkey`: a single key (`["enter"]`) or a chord (`["ctrl", "c"]`).

  • scroll_xinteger· int64

    Horizontal scroll amount for `action=scroll`. Default 0.

  • scroll_yinteger· int64

    Vertical scroll amount for `action=scroll`; positive scrolls down. Default 3.

  • session_idstringrequired
  • textstring

    Text to type, for `action=type`.

  • to_xinteger· int64

    Drag end x (with `to_y`), for `action=drag`.

  • to_yinteger· int64

    Drag end y.

  • xinteger· int64

    Target x (pixels). Required for click/right_click/double_click/move and the drag start; the scroll anchor defaults to screen center.

  • yinteger· int64

    Target y (pixels).

Response
  • detailstringrequired

    What was done, for the transcript.

  • okbooleanrequired

computer::displays

Function

List the local displays (index, name, primary, size). Pass the chosen index as `monitor` to computer::sessions::start to drive that display; omit it to use the display under the cursor. Native host only.

Request
Empty object
Response
  • displaysobject[]required
    • builtinbooleanrequired
    • heightinteger· uint32requiredmin 0

      Logical height in points.

    • indexinteger· uint32requiredmin 0

      Index to pass as `monitor` to `computer::sessions::start`.

    • namestringrequired
    • primarybooleanrequired
    • widthinteger· uint32requiredmin 0

      Logical width in points.

computer::frame

Function

Internal: newest screencast frame, or nothing when since_frame is still current. No capture round-trip; poll fast. Not an agent function.

Request
  • session_idstringrequired
  • since_frameinteger· uint64min 0

    Frame cursor from the previous read; when the newest frame still has this seq the response omits `frame` (nothing changed).

Response
  • activebooleanrequired

    False when no screencast is running (call screencast::start first).

  • framestring

    Base64 image of the newest frame; absent when `since_frame` is still current or no frame has arrived yet.

  • frame_seqinteger· uint64requiredmin 0
  • heightinteger· uint32requiredmin 0
  • mimestringrequired

    Image mime of the frame bytes.

  • timestampinteger· int64required
  • widthinteger· uint32requiredmin 0

computer::observe

Function

Capture the desktop plus, optionally, the accessibility tree. Use include_a11y on macOS guests for a machine-readable element tree; elsewhere prefer computer::screenshot.

Request
  • include_a11yboolean

    Also fetch the accessibility tree. macOS returns a real tree; other guests may return a stub or nothing (the field is then omitted).

  • session_idstringrequired
Response
  • accessibilityunknown

    Accessibility tree, present only when requested and the guest exposes one.

  • contentobject[]required
    • datastring
    • mimestring
    • textstring
    • typestringrequired
  • detailsobjectrequired
    • mimestringrequired
    • screenobjectrequired

      Desktop pixel dimensions. Coordinates handed to pointer actions are in this space: integer pixels, top-left origin, 1:1 with the screenshot.

      • heightinteger· uint32requiredmin 0
      • widthinteger· uint32requiredmin 0
    • session_idstringrequired

computer::on-config-change

Function

Internal: reload computer settings from the authoritative configuration on change.

Request
Empty object
Response
  • okbooleanrequired

computer::screencast::start

Function

Internal: start pushing live desktop frames onto the computer:frames stream for the console viewport. Console-UI plumbing; agents use computer::screenshot. Not an agent function.

Request
  • session_idstringrequired
Response
  • okbooleanrequired

computer::screencast::stop

Function

Internal: stop the live frame push. Idempotent. Not an agent function.

Request
  • session_idstringrequired
Response
  • okbooleanrequired

computer::screenshot

Function

Capture the desktop as a viewable image. This is how you see the screen before acting; the coordinate space of computer::act is this image's pixels (top-left origin).

Request
  • session_idstringrequired
Response
  • contentobject[]required
    • datastring
    • mimestring
    • textstring
    • typestringrequired
  • detailsobjectrequired
    • heightinteger· uint32requiredmin 0
    • mimestringrequired

      Detected image mime (`image/png` or `image/jpeg`).

    • session_idstringrequired
    • widthinteger· uint32requiredmin 0

computer::sessions::list

Function

List live computer sessions with their endpoint, guest OS, and screen size.

Request
Empty object
Response
  • sessionsobject[]required
    • created_msinteger· int64required
    • endpointstringrequired
    • last_used_msinteger· int64required
    • osstringrequired
    • screenobjectrequired

      Desktop pixel dimensions. Coordinates handed to pointer actions are in this space: integer pixels, top-left origin, 1:1 with the screenshot.

      • heightinteger· uint32requiredmin 0
      • widthinteger· uint32requiredmin 0
    • screencast_activebooleanrequired

      True while a live screen stream is running for this session.

    • session_idstringrequired

computer::sessions::start

Function

Start a computer-use session and return its session_id. Pass `image` to boot a fresh desktop in an iii-sandbox microVM (fixed virtual display, no host setup), an `endpoint` to drive a desktop through its guest executor, or omit both to drive the local machine. Sessions are durable; stop them with computer::sessions::stop when done.

Request
  • endpointstring

    Desktop to drive when not using a sandbox `image`. Omit (and leave `image` unset) to drive the local machine this worker runs on (native driver, nothing else to run). Pass the endpoint of a desktop guest's executor (a `ws`/`wss`/`http`/`https` url or a bare `host:port`) to drive a remote desktop; falls back to the configured `default_endpoint` when omitted.

  • imagestring

    Boot a fresh desktop inside an iii-sandbox microVM from this OCI image (a sandbox preset name or `custom_images` key) and drive it through iii primitives alone. A fixed virtual display means 1:1 coordinates, no HiDPI or multi-monitor ambiguity. Falls back to the configured `sandbox_image` when omitted. Takes precedence over `endpoint`.

  • monitorinteger· uint32min 0

    Display index (from `computer::displays`) for a native session. Omit to drive the display under the cursor. Ignored for a remote `endpoint` or a sandbox `image`.

  • osstring

    Guest OS label recorded on the session and surfaced in `session-started` (`linux`, `macos`, `windows`, `android`). Omit and each driver labels itself: a sandbox session is `linux`, a remote one takes the configured `os`, and a native one takes this host's OS.

Response
  • endpointstringrequired

    What the session drives: `native` for the local machine, or the normalized remote endpoint.

  • osstringrequired
  • screenall ofrequired

    Desktop pixel dimensions; the coordinate space for `computer::act`.

    all of (1)
    Variant 1
    • heightinteger· uint32requiredmin 0
    • widthinteger· uint32requiredmin 0
  • session_idstringrequired

    Pass this to every other computer function.

computer::sessions::stop

Function

Stop a computer session and close its driver connection. Idempotent: stopping an unknown or already-stopped session succeeds with was_running=false.

Request
  • session_idstringrequired

    Session to stop. Stopping an unknown or already-stopped id succeeds.

Response
  • okbooleanrequired
  • was_runningbooleanrequired

    False when the session was already gone.

computer::ui-content

Function

Serve the computer worker's injected console UI assets (content function for its console:script / console:style triggers).

Request
  • pathstringrequired

    The asset path from the trigger config (e.g. `state/page.js`).

Response
  • contentstringrequired

    The asset source, verbatim.

  • content_typestringrequired

    MIME type the console should serve the asset with.

Triggers2

computer::session-started

Trigger

A desktop session connected and is ready to drive.

Invocation
  • session_idstring

    Only deliver events for this computer session.

Return
valueunknown

computer::session-stopped

Trigger

A desktop session ended (stopped or idle).

Invocation
  • session_idstring

    Only deliver events for this computer session.

Return
valueunknown

Search the registry

Find workers, kits and executables, or jump to a page.