web

Outbound HTTP(S) client on the iii engine bus. A single bus function, web::fetch, performs a network request and returns a parsed { ok, status, headers, body } envelope with enforced size/timeout caps and server-side SSRF protection. It is the network counterpart to the shell worker: use web::fetch instead of shell::exec with curl/wget for any HTTP request.

The worker is a faithful Rust port of the original TypeScript web::fetch implementation — same request fields and success/error/image envelopes, so existing callers and the harness consumer are unaffected.

While connected it also injects a usage section into the agent system prompt via the harness pre-generate hook (web::inject-guidance), so the guidance is presence-gated: no web worker, no prompt text. The binding is one-shot at startup and relies on the engine's recoverable triggers (iii #1962, engine ≥ 0.21.8): bound before the harness is up, it parks as a pending intent and activates when the harness registers the trigger type. On older engines the bind is silently dropped.

Table of contents

1. Install
2. Configuration
3. The web::fetch function
4. Reading the result
5. Page-reading mode
6. SSRF guard
7. Local development & testing

---

Install

iii worker add web

iii worker add fetches the binary (web), writes a config block into ~/.iii/config.yaml, and the engine starts the worker on the next iii start.

---

Configuration

The worker reads its operational ceilings from the configuration worker under the web key. It registers a schema + seed on boot, fetches the authoritative value, and hot-reloads on change. All keys are optional and fall back to the defaults below.

web:
  default_timeout_ms: 30000      # per-request timeout when caller omits timeout_ms
  max_timeout_ms: 120000         # ceiling; caller timeout_ms is clamped DOWN to this
  default_response_bytes: 262144 # default body cap in page-reading mode (256 KiB)
  max_response_bytes: 5242880    # absolute body ceiling (5 MiB); caller max_bytes clamped to this
  max_transform_bytes: 1048576   # HTML→markdown/text runs only on bodies ≤ this (1 MiB)
  max_redirects: 5               # hops before too_many_redirects
  user_agent: 'iii-harness/0.1 (+web::fetch)'
  allow_loopback: true           # set false in prod so 127.0.0.0/8, ::1, localhost are blocked

A copy of these defaults lives in config.yaml.example.

---

The web::fetch function

Minimal call — a URL is the only required field; everything else has a default:

// agent_trigger { function: "web::fetch", payload: { "url": "https://api.github.com/zen" } }
// → { "ok": true, "status": 200, "body": "Design for failure.", ... }

Request fields

Field	Default	Notes
url (required)	—	absolute http(s)://
method	GET	case-insensitive; GET HEAD POST PUT PATCH DELETE OPTIONS
headers	{}	object; keys as written
json	—	structured payload; auto-stringified + sets content-type: application/json. Wins over body.
body	—	raw string body; ignored on GET/HEAD
response_format	"text"	"text" | "base64" (binary) | "json" (also parses into json)
format	—	page-reading mode: "markdown" | "text" | "html" (see below)
content_filter	—	{ type: "pruning"|"bm25", query?, threshold?, threshold_type?, min_word_threshold? }. Page mode only. When set, body holds the filtered content (pruning default threshold 0.48; bm25 default 1.0; query falls back to page metadata).
target_elements	—	CSS selectors (tl subset: tag/.class/#id); restrict rendered content to these regions
excluded_tags	—	tag/selectors to drop before rendering (e.g. nav, footer)
include_links	false	adds links: { internal, external } (absolute URLs, classified by host)
include_media	false	adds media: { images, videos, audios } (absolute URLs)
timeout_ms	default_timeout_ms	clamped DOWN to max_timeout_ms
max_bytes	max_response_bytes (default_response_bytes in format mode)	over-cap body is truncated, not errored
follow_redirects	true	each hop re-checked against the SSRF blocklist

Response

Success (ok: true) — returned for any completed response, 2xx through 5xx:

{
  "ok": true,
  "status": 200,
  "status_text": "OK",
  "headers": { "content-type": "application/json", ... },  // keys LOWER-CASED
  "body": "<utf8 text | base64 when response_format=base64>",
  "json": { ... },            // only when response_format="json" and parse succeeded
  "parse_error": "…",         // only when response_format="json" and JSON.parse failed
  "response_format": "json",
  "bytes_truncated": false,   // true when body hit max_bytes (NOT an error)
  "redirect_chain": ["https://…/a"],  // omitted when no redirects
  "content_type": "text/html",  // page-reading mode only
  "transformed": "markdown",    // only when an HTML transform actually ran
  "links": { "internal": [{ "href": "…", "text": "…" }], "external": [ … ] },  // include_links
  "media": { "images": [{ "src": "…", "alt": "…" }], "videos": [ … ], "audios": [ … ] },  // include_media
}

Failure (ok: false) — the fetch never produced a response (no status):

{ "ok": false, "error": "blocked_host", "message": "…" }

error	Cause	Fix
invalid_payload	Payload failed schema (bad method, wrong types)	Correct the named fields
invalid_url	url isn't a parseable absolute http(s):// URL	Pass a full absolute URL incl. scheme
blocked_host	Target resolves to a private / link-local / cloud-metadata IP	Don't target internal/metadata hosts; for dev loopback set allow_loopback
timeout	Slower than timeout_ms	Raise timeout_ms (up to ceiling) or shrink work via max_bytes
too_many_redirects	More than max_redirects hops	Use the final URL, or follow_redirects: false and read the location header
transport_error	Connection refused/reset, TLS/DNS failure, or unparseable redirect Location	Check host/port/cert; retry if transient

There is no too_large error — oversize responses come back ok: true with bytes_truncated: true.

---

Reading the result

ok tells you if the request COMPLETED, not whether the server was happy.

• A 404 or 500 is a successful fetch → ok: true, status: 404. Branch on status for HTTP outcomes.
• ok: false means the request never produced a response. Branch on error.

if (!r.ok)              → fetch failed; look at r.error
else if (r.status>=400) → server returned an HTTP error; look at r.status / r.body
else                    → success; use r.body or r.json

---

Page-reading mode

response_format (transport encoding) and format (page-reading) are orthogonal — set one, not both. When format is set, response_format is ignored.

• format: "markdown" — text/html → Markdown (the right default for reading pages; far fewer tokens than raw HTML).
• format: "text" — text/html → plain text.
• format: "html" — raw HTML.

When content_filter is set, body is the filtered output (there is no separate field) — feed it straight to a model. (If filtering would empty the page, or the page is too large or deeply nested to transform, body falls back to the unfiltered content.)

In page-reading mode the request goes out with a browser User-Agent + format-matched Accept header, and retries once with the honest UA on a Cloudflare cf-mitigated: challenge response. The HTML→markdown/text transform runs only on bodies ≤ max_transform_bytes; larger pages come back raw with transformed unset.

Images: a viewable image/* response (jpeg/png/gif/webp, 2xx, complete, non-empty) returns the actual image plus a one-line text summary instead of the JSON envelope. Anything else falls back to the normal envelope with response_format: "base64".

---

SSRF guard

DNS is resolved once, every resolved address is checked, then the request is dialed at the validated IP (pinned lookup + TLS servername) — so the IP that passed the check is the IP connected to (no DNS-rebind window). On each 3xx the Location is re-resolved and re-validated before following, and Authorization / Cookie / Proxy-Authorization are stripped when the redirect changes host or downgrades https → http.

Blocked unconditionally → error: "blocked_host":

• Private RFC1918 (10/8, 172.16/12, 192.168/16)
• Link-local incl. cloud metadata 169.254.169.254
• IPv6 ULA (fc00::/7), link-local (fe80::/10), multicast
• ::ffff:-mapped IPv4 in both dotted (::ffff:169.254.169.254) and hex (::ffff:a9fe:a9fe) forms

Loopback (127.0.0.0/8, ::1, localhost) is allowed by default for dev convenience; operators set web.allow_loopback: false for prod, after which loopback returns blocked_host.

---

Local development & testing

cargo build                  # build the web binary
cargo test                   # unit + wiremock integration + adversarial SSRF tests
cargo clippy -- -D warnings  # lints
cargo fmt --check            # formatting

The agent-facing authoring guide lives in skills/index.md (served at runtime via directory::skills::get). For the exact, authoritative field types, call engine::functions::info { function_id: "web::fetch" } — the live schema wins if it ever disagrees with this document.