browser

Interactive Chromium sessions on the iii engine bus. Agents start a session, read the page as an accessibility-tree outline, click and type against element refs, and read the page's own console and network history back as data. The single most important thing it gives you: "why is my dev server page blank?" becomes answerable, because the page's console errors are one browser::console::read away. The console worker adds the human window: a live Browser page with a streaming viewport (Chromium-pushed screencast frames), the console feed, and click-to-pick elements into chat.

In the console

An agent reads a page as an accessibility outline (browser::snapshot) while you watch the live viewport and console feed:

browser::screenshot renders the captured image inline in the chat card:

Pick mode highlights the element under the cursor and drops it into the chat composer as an actionable ref:

Install

iii worker add browser

iii worker add fetches the binary, writes a config block into ~/.iii/config.yaml, and the engine starts the worker on the next iii start. The worker drives a Chromium/Chrome already installed on the machine; point executable at a specific binary if auto-detection picks the wrong one.

To watch sessions live, pick elements into chat, and follow the agent's browsing from a UI, add the console worker as well:

iii worker add console

Quickstart

Start a session, read the page, act on it, then read the console:

use iii_sdk::protocol::TriggerRequest;
use iii_sdk::{register_worker, InitOptions};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let iii = register_worker("ws://localhost:49134", InitOptions::default());

    let started = iii.trigger(TriggerRequest {
        function_id: "browser::sessions::start".into(),
        payload: json!({ "url": "http://localhost:3000" }),
        action: None,
        timeout_ms: Some(30_000),
    }).await?;
    let session_id = started["session_id"].as_str().unwrap();

    // The page as text: an a11y outline with [ref=eN] handles.
    let snapshot = iii.trigger(TriggerRequest {
        function_id: "browser::snapshot".into(),
        payload: json!({ "session_id": session_id }),
        action: None,
        timeout_ms: Some(15_000),
    }).await?;
    println!("{}", snapshot["tree"].as_str().unwrap());

    // What did the page log? Errors only, no dump.
    let console = iii.trigger(TriggerRequest {
        function_id: "browser::console::read".into(),
        payload: json!({ "session_id": session_id, "level": "error" }),
        action: None,
        timeout_ms: Some(10_000),
    }).await?;
    println!("{console:#}");
    Ok(())
}

The rest of the surface: browser::act (click/hover/type/press/scroll by ref or coordinates, left/right/middle and double-click), browser::evaluate (JS expression), browser::screenshot (viewable JPEG), browser::history (back/forward/reload), browser::network::read (requests + failures), browser::dom::read (DOM tree with refs), browser::styles::read / browser::styles::write (computed styles + live inline edits, the design panel backing), and browser::sessions::list / browser::sessions::stop. Function ids and schemas live in the code and iii worker info browser.

Beyond single actions: browser::execute runs a multi-step async script in the page — top-level await, log(...), sleep(ms), waitFor(selector), and a state object that persists across execute calls for the session — so one call replaces a chain of act/evaluate round-trips. browser::snapshot accepts diff: true to return only what changed since the previous snapshot, and reports the document generation its refs belong to (ref names are unique per snapshot and fail closed when stale, never resolving to a different element). browser::sessions::start accepts read_only: true for inspection-only sessions where act/evaluate/execute/styles::write are rejected. browser::doctor reports the environment — detected Chromium, version, capacity — with an enable_how string for anything degraded.

browser::sessions::attach binds a session to an already-running browser over CDP (start Chrome with --remote-debugging-port) instead of launching one, so it reaches the real profile with its logins and extensions. It opens a fresh tab the session owns, or adopts an existing tab by URL substring and releases it untouched on stop; browser::tabs::list enumerates a running browser's tabs. Attach reaches logged-in state, so it is off unless allow_attach is set in config, and adoption is exclusive per tab.

browser::handoff pauses a session for a step only a human can do (CAPTCHA, 2FA, payment): it mounts an in-page continue banner and blocks the call until the human clicks it, a browser::handoff::confirm call resolves it, or the timeout elapses, emitting browser::handoff-requested for the console to surface. Human acknowledgment is not proof, so the caller verifies the expected page state after it returns.

browser::recording::start / browser::recording::stop capture a session's live viewport to a webm or mp4 file by piping the screencast through ffmpeg (turning screencast on if needed); stop returns the path, duration, and frame count. While screencast is active a human watching the viewport also sees a ghost cursor following the agent's clicks and a session-status badge; both are fixed-position in-page overlays that never touch page content. browser::doctor reports whether ffmpeg (recording) and attach mode are available.

Configuration

Stored in the configuration worker under the browser key; every field is editable live from the console. Caps and timeouts hot-reload; executable, headless, and the viewport apply to sessions started after the change.

browser:
  executable: ''            # empty = auto-detect Chrome/Chromium/Edge
  user_data_dir: ''         # set a path to persist cookies/logins across sessions
  headless: true            # false shows a real window locally
  max_sessions: 4           # concurrent Chromium processes
  console_buffer: 500       # per-session console ring buffer (entries)
  network_buffer: 500       # per-session network ring buffer (entries)
  viewport_width: 1280
  viewport_height: 800
  default_timeout_ms: 30000 # navigation/act/evaluate default
  max_timeout_ms: 120000    # ceiling; caller timeout_ms clamped DOWN to this
  idle_stop_ms: 300000      # stop sessions idle this long; 0 disables
  screenshot_quality: 60    # JPEG quality 1-100
  allowed_schemes: [http, https]
  max_snapshot_nodes: 2000  # a11y outline size cap
  allow_attach: false       # true = allow sessions::attach into a running browser's real profile

Custom trigger types

Sibling workers (and the console UI) can subscribe to session activity. All bindings accept an optional { "session_id": "..." } filter.

Trigger type	Fires when	Payload to subscribers
browser::session-started	A session is up and ready	{ session_id, url, headless, timestamp }
browser::session-stopped	A session ended	{ session_id, reason: "stopped" | "idle" | "crashed", timestamp }
browser::navigated	The page committed a navigation	{ session_id, url, timestamp }
browser::console-event	A console/log/exception entry was captured	{ session_id, entry }
browser::picked	The human picked an element in inspect mode	{ session_id, element, timestamp }
browser::handoff-requested	A session paused for a human step (CAPTCHA, 2FA, payment)	{ session_id, handoff_id, instructions, timestamp }

browser::console-event is high-volume; bind it with a session_id filter and treat browser::console::read as the durable record. browser::picked elements carry a ref that browser::act accepts directly, so a human pick flows straight into agent action.

Element picking

browser::pick::start puts the page in DevTools inspect mode (native hover highlight); the human's click resolves to tag, attributes, outer HTML, text, bounds, and recent console errors, emitted as browser::picked. The pick, hint, screencast, and frame functions are internal: console-UI plumbing, not agent surface, and they stay out of agent tool lists.