web

Outbound HTTP(S) client on the iii engine bus. A single bus function, web::fetch, performs a network request and returns a parsed { ok, status, headers, body } envelope with enforced size/timeout caps and server-side SSRF protection. It is the network counterpart to the shell worker: use web::fetch instead of shell::exec with curl/wget for any HTTP request.

The worker is a faithful Rust port of the original TypeScript web::fetch implementation — same request fields and success/error/image envelopes, so existing callers and the harness consumer are unaffected.

While connected (and unless turned off) it also injects a usage section into the agent system prompt via the harness pre-generate hook (web::inject-guidance). inject_guidance in the web configuration entry is ON by default; turning it off (the Console's global Settings modal, or configuration::set) hot-applies with no restart, and the harness's # Granted functions catalog still advertises web::fetch. The guidance is presence-gated: no web worker, no prompt text. The binding is one-shot at startup and relies on the engine's recoverable triggers (iii #1962, engine ≥ 0.21.8): bound before the harness is up, it parks as a pending intent and activates when the harness registers the trigger type. On older engines the bind is silently dropped.

Table of contents

1. Install
2. Configuration
3. The web::fetch function
4. Reading the result
5. Page-reading mode
6. SSRF guard
7. Local development & testing

Install

iii trigger compose::add worker=web

iii trigger compose::add resolves the worker and its dependencies, writes exact declarations to worker-compose.yaml, and reconciles the Compose project.

Configuration

The worker reads its operational ceilings from the configuration worker under the web key. It registers a schema + seed on boot, fetches the authoritative value, and hot-reloads on change. All keys are optional and fall back to the defaults below.

web:
  default_timeout_ms: 30000      # per-request timeout when caller omits timeout_ms
  max_timeout_ms: 120000         # ceiling; caller timeout_ms is clamped DOWN to this
  default_response_bytes: 262144 # default body cap in page-reading mode (256 KiB)
  max_response_bytes: 5242880    # absolute body ceiling (5 MiB); caller max_bytes clamped to this
  max_transform_bytes: 1048576   # HTML→markdown/text runs only on bodies ≤ this (1 MiB)
  max_redirects: 5               # hops before too_many_redirects
  user_agent: 'iii-harness/0.1 (+web::fetch)'
  allow_loopback: true           # set false in prod so 127.0.0.0/8, ::1, localhost are blocked

A copy of these defaults lives in config.yaml.example.

The web::fetch function

Minimal call — a URL is the only required field; everything else has a default:

// agent_trigger { function: "web::fetch", payload: { "url": "https://api.github.com/zen" } }
// → { "ok": true, "status": 200, "body": "Design for failure.", ... }

Request fields

Response

Success (ok: true) — returned for any completed response, 2xx through 5xx:

{
  "ok": true,
  "status": 200,
  "status_text": "OK",
  "headers": { "content-type": "application/json", ... },  // keys LOWER-CASED
  "body": "<utf8 text | base64 when response_format=base64>",
  "json": { ... },            // only when response_format="json" and parse succeeded
  "parse_error": "…",         // only when response_format="json" and JSON.parse failed
  "response_format": "json",
  "bytes_truncated": false,   // true when body hit max_bytes (NOT an error)
  "redirect_chain": ["https://…/a"],  // omitted when no redirects
  "content_type": "text/html",  // page-reading mode only
  "transformed": "markdown",    // only when an HTML transform actually ran
  "links": { "internal": [{ "href": "…", "text": "…" }], "external": [ … ] },  // include_links
  "media": { "images": [{ "src": "…", "alt": "…" }], "videos": [ … ], "audios": [ … ] },  // include_media
}

Failure (ok: false) — the fetch never produced a response (no status):

{ "ok": false, "error": "blocked_host", "message": "…" }

There is no too_large error — oversize responses come back ok: true with bytes_truncated: true.

Reading the result

ok tells you if the request COMPLETED, not whether the server was happy.

• A 404 or 500 is a successful fetch → ok: true, status: 404. Branch on status for HTTP outcomes.
• ok: false means the request never produced a response. Branch on error.

if (!r.ok)              → fetch failed; look at r.error
else if (r.status>=400) → server returned an HTTP error; look at r.status / r.body
else                    → success; use r.body or r.json

Page-reading mode

response_format (transport encoding) and format (page-reading) are orthogonal — set one, not both. When format is set, response_format is ignored.

• format: "markdown" — text/html → Markdown (the right default for reading pages; far fewer tokens than raw HTML).
• format: "text" — text/html → plain text.
• format: "html" — raw HTML.

When content_filter is set, body is the filtered output (there is no separate field) — feed it straight to a model. (If filtering would empty the page, or the page is too large or deeply nested to transform, body falls back to the unfiltered content.)

In page-reading mode the request goes out with a browser User-Agent + format-matched Accept header, and retries once with the honest UA on a Cloudflare cf-mitigated: challenge response. The HTML→markdown/text transform runs only on bodies ≤ max_transform_bytes; larger pages come back raw with transformed unset.

Images: a viewable image/* response (jpeg/png/gif/webp, 2xx, complete, non-empty) returns the actual image plus a one-line text summary instead of the JSON envelope. Anything else falls back to the normal envelope with response_format: "base64".

SSRF guard

DNS is resolved once, every resolved address is checked, then the request is dialed at the validated IP (pinned lookup + TLS servername) — so the IP that passed the check is the IP connected to (no DNS-rebind window). On each 3xx the Location is re-resolved and re-validated before following, and Authorization / Cookie / Proxy-Authorization are stripped when the redirect changes host or downgrades https → http.

Blocked unconditionally → error: "blocked_host":

• Private RFC1918 (10/8, 172.16/12, 192.168/16)
• Link-local incl. cloud metadata 169.254.169.254
• IPv6 ULA (fc00::/7), link-local (fe80::/10), multicast
• ::ffff:-mapped IPv4 in both dotted (::ffff:169.254.169.254) and hex (::ffff:a9fe:a9fe) forms

Loopback (127.0.0.0/8, ::1, localhost) is allowed by default for dev convenience; operators set web.allow_loopback: false for prod, after which loopback returns blocked_host.

Local development & testing

cargo build                  # build the web binary
cargo test                   # unit + wiremock integration + adversarial SSRF tests
cargo clippy -- -D warnings  # lints
cargo fmt --check            # formatting

The agent-facing authoring guide lives in skills/SKILL.md (served at runtime via directory::skills::get). For the exact, authoritative field types, call engine::functions::info { function_id: "web::fetch" } — the live schema wins if it ever disagrees with this document.