web
v1.2.3Outbound HTTP client on the iii bus (web::fetch).
- macOS: arm64 · x64
- Linux: arm64 · armv7 · x64
- Windows: arm64 · x64 · x86
full markdown
/workers/web.md. paste it into an llm prompt or pipe it through curl from a worker.install
dependencies
readme
web
Outbound HTTP(S) client on the iii engine
bus. A single bus function, web::fetch, performs a network request and
returns a parsed { ok, status, headers, body } envelope with enforced
size/timeout caps and server-side SSRF protection. It is the network
counterpart to the shell worker: use web::fetch instead of
shell::exec with curl/wget for any HTTP request.
The worker is a faithful Rust port of the original TypeScript web::fetch
implementation — same request fields and success/error/image envelopes, so
existing callers and the harness consumer are unaffected.
While connected it also injects a usage section into the agent system prompt
via the harness pre-generate hook (web::inject-guidance), so the guidance
is presence-gated: no web worker, no prompt text. The binding is one-shot at
startup and relies on the engine's recoverable triggers (iii #1962, engine ≥
0.21.8): bound before the harness is up, it parks as a pending intent and
activates when the harness registers the trigger type. On older engines the
bind is silently dropped.
Table of contents
- Install
- Configuration
- The
web::fetchfunction - Reading the result
- Page-reading mode
- SSRF guard
- Local development & testing
Install
iii worker add webiii worker add fetches the binary (web), writes a config block into
~/.iii/config.yaml, and the engine starts the worker on the next
iii start.
Configuration
The worker reads its operational ceilings from the configuration worker
under the web key. It registers a schema + seed on boot, fetches the
authoritative value, and hot-reloads on change. All keys are optional and
fall back to the defaults below.
web:
default_timeout_ms: 30000 # per-request timeout when caller omits timeout_ms
max_timeout_ms: 120000 # ceiling; caller timeout_ms is clamped DOWN to this
default_response_bytes: 262144 # default body cap in page-reading mode (256 KiB)
max_response_bytes: 5242880 # absolute body ceiling (5 MiB); caller max_bytes clamped to this
max_transform_bytes: 1048576 # HTML→markdown/text runs only on bodies ≤ this (1 MiB)
max_redirects: 5 # hops before too_many_redirects
user_agent: 'iii-harness/0.1 (+web::fetch)'
allow_loopback: true # set false in prod so 127.0.0.0/8, ::1, localhost are blockedA copy of these defaults lives in config.yaml.example.
The web::fetch function
Minimal call — a URL is the only required field; everything else has a default:
// agent_trigger { function: "web::fetch", payload: { "url": "https://api.github.com/zen" } }
// → { "ok": true, "status": 200, "body": "Design for failure.", ... }Request fields
| Field | Default | Notes |
|---|---|---|
url (required) |
— | absolute http(s):// |
method |
GET |
case-insensitive; GET HEAD POST PUT PATCH DELETE OPTIONS |
headers |
{} |
object; keys as written |
json |
— | structured payload; auto-stringified + sets content-type: application/json. Wins over body. |
body |
— | raw string body; ignored on GET/HEAD |
response_format |
"text" |
"text" | "base64" (binary) | "json" (also parses into json) |
format |
— | page-reading mode: "markdown" | "text" | "html" (see below) |
content_filter |
— | { type: "pruning"|"bm25", query?, threshold?, threshold_type?, min_word_threshold? }. Page mode only. When set, body holds the filtered content (pruning default threshold 0.48; bm25 default 1.0; query falls back to page metadata). |
target_elements |
— | CSS selectors (tl subset: tag/.class/#id); restrict rendered content to these regions |
excluded_tags |
— | tag/selectors to drop before rendering (e.g. nav, footer) |
include_links |
false |
adds links: { internal, external } (absolute URLs, classified by host) |
include_media |
false |
adds media: { images, videos, audios } (absolute URLs) |
timeout_ms |
default_timeout_ms |
clamped DOWN to max_timeout_ms |
max_bytes |
max_response_bytes (default_response_bytes in format mode) |
over-cap body is truncated, not errored |
follow_redirects |
true |
each hop re-checked against the SSRF blocklist |
Response
Success (ok: true) — returned for any completed response, 2xx through 5xx:
{
"ok": true,
"status": 200,
"status_text": "OK",
"headers": { "content-type": "application/json", ... }, // keys LOWER-CASED
"body": "<utf8 text | base64 when response_format=base64>",
"json": { ... }, // only when response_format="json" and parse succeeded
"parse_error": "…", // only when response_format="json" and JSON.parse failed
"response_format": "json",
"bytes_truncated": false, // true when body hit max_bytes (NOT an error)
"redirect_chain": ["https://…/a"], // omitted when no redirects
"content_type": "text/html", // page-reading mode only
"transformed": "markdown", // only when an HTML transform actually ran
"links": { "internal": [{ "href": "…", "text": "…" }], "external": [ … ] }, // include_links
"media": { "images": [{ "src": "…", "alt": "…" }], "videos": [ … ], "audios": [ … ] }, // include_media
}Failure (ok: false) — the fetch never produced a response (no status):
{ "ok": false, "error": "blocked_host", "message": "…" }error |
Cause | Fix |
|---|---|---|
invalid_payload |
Payload failed schema (bad method, wrong types) |
Correct the named fields |
invalid_url |
url isn't a parseable absolute http(s):// URL |
Pass a full absolute URL incl. scheme |
blocked_host |
Target resolves to a private / link-local / cloud-metadata IP | Don't target internal/metadata hosts; for dev loopback set allow_loopback |
timeout |
Slower than timeout_ms |
Raise timeout_ms (up to ceiling) or shrink work via max_bytes |
too_many_redirects |
More than max_redirects hops |
Use the final URL, or follow_redirects: false and read the location header |
transport_error |
Connection refused/reset, TLS/DNS failure, or unparseable redirect Location |
Check host/port/cert; retry if transient |
There is no too_large error — oversize responses come back ok: true
with bytes_truncated: true.
Reading the result
ok tells you if the request COMPLETED, not whether the server was happy.
- A 404 or 500 is a successful fetch →
ok: true,status: 404. Branch onstatusfor HTTP outcomes. ok: falsemeans the request never produced a response. Branch onerror.
if (!r.ok) → fetch failed; look at r.error
else if (r.status>=400) → server returned an HTTP error; look at r.status / r.body
else → success; use r.body or r.jsonPage-reading mode
response_format (transport encoding) and format (page-reading) are
orthogonal — set one, not both. When format is set, response_format is
ignored.
format: "markdown"—text/html→ Markdown (the right default for reading pages; far fewer tokens than raw HTML).format: "text"—text/html→ plain text.format: "html"— raw HTML.
When content_filter is set, body is the filtered output (there is no separate field) — feed it straight to a model. (If filtering would empty the page, or the page is too large or deeply nested to transform, body falls back to the unfiltered content.)
In page-reading mode the request goes out with a browser User-Agent +
format-matched Accept header, and retries once with the honest UA on a
Cloudflare cf-mitigated: challenge response. The HTML→markdown/text
transform runs only on bodies ≤ max_transform_bytes; larger pages come
back raw with transformed unset.
Images: a viewable image/* response (jpeg/png/gif/webp, 2xx,
complete, non-empty) returns the actual image plus a one-line text summary
instead of the JSON envelope. Anything else falls back to the normal
envelope with response_format: "base64".
SSRF guard
DNS is resolved once, every resolved address is checked, then the
request is dialed at the validated IP (pinned lookup + TLS
servername) — so the IP that passed the check is the IP connected to (no
DNS-rebind window). On each 3xx the Location is re-resolved and
re-validated before following, and Authorization / Cookie /
Proxy-Authorization are stripped when the redirect changes host or
downgrades https → http.
Blocked unconditionally → error: "blocked_host":
- Private RFC1918 (
10/8,172.16/12,192.168/16) - Link-local incl. cloud metadata
169.254.169.254 - IPv6 ULA (
fc00::/7), link-local (fe80::/10), multicast ::ffff:-mapped IPv4 in both dotted (::ffff:169.254.169.254) and hex (::ffff:a9fe:a9fe) forms
Loopback (127.0.0.0/8, ::1, localhost) is allowed by default for
dev convenience; operators set web.allow_loopback: false for prod, after
which loopback returns blocked_host.
Local development & testing
cargo build # build the web binary
cargo test # unit + wiremock integration + adversarial SSRF tests
cargo clippy -- -D warnings # lints
cargo fmt --check # formattingThe agent-facing authoring guide lives in
skills/index.md (served at runtime via
directory::skills::get). For the exact, authoritative field types, call
engine::functions::info { function_id: "web::fetch" } — the live schema
wins if it ever disagrees with this document.
api reference (json)
{
"functions": [
{
"description": "Fetch a URL over HTTP(S) and return the response as a structured envelope. Use this INSTEAD of `shell::exec` with curl for any HTTP request — it returns {ok, status, headers, body} as JSON, enforces size/timeout caps, and blocks private / cloud-metadata / link-local addresses server-side (SSRF guard; loopback is allowed by default for harness dev workflows). To READ A WEB PAGE, set `format: \"markdown\"` — HTML is converted to Markdown and images come back viewable (image responses in that mode return the image itself plus a text line, not the {ok,...} envelope). For JSON: pass `json: {...}` (auto-stringifies + sets content-type) and `response_format: \"json\"` (auto-parses response into the `json` field). Method is case-insensitive. On failure returns `{ok:false, error, message}` where `error` is one of: invalid_payload, invalid_url, blocked_host, timeout, too_many_redirects, transport_error. Branch on `error`, not on text.",
"metadata": {},
"name": "web::fetch",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ContentFilter": {
"description": "Content-filter request block. `query`, `threshold`, `threshold_type`, and `min_word_threshold` are optional; their default values are applied when the filter runs (see the content pipeline), not at deserialization.",
"properties": {
"min_word_threshold": {
"default": null,
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"threshold": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"threshold_type": {
"anyOf": [
{
"$ref": "#/definitions/ThresholdType"
},
{
"type": "null"
}
]
},
"type": {
"$ref": "#/definitions/FilterType"
}
},
"required": [
"type"
],
"type": "object"
},
"FilterType": {
"enum": [
"pruning",
"bm25"
],
"type": "string"
},
"PageFormat": {
"enum": [
"markdown",
"text",
"html"
],
"type": "string"
},
"ResponseFormat": {
"enum": [
"text",
"base64",
"json"
],
"type": "string"
},
"ThresholdType": {
"enum": [
"fixed",
"dynamic"
],
"type": "string"
}
},
"properties": {
"body": {
"default": null,
"description": "Raw request body as a string. Prefer `json` for JSON payloads.",
"type": [
"string",
"null"
]
},
"content_filter": {
"anyOf": [
{
"$ref": "#/definitions/ContentFilter"
},
{
"type": "null"
}
],
"description": "Content filter: prunes boilerplate. When set, `body` holds the filtered output in the requested `format` (page-reading mode only)."
},
"excluded_tags": {
"default": null,
"description": "Tag names / selectors to drop before rendering (e.g. \"nav\", \"footer\").",
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"follow_redirects": {
"default": null,
"description": "Follow 3xx redirects. Defaults to true. Each hop is re-checked against the SSRF blocklist.",
"type": [
"boolean",
"null"
]
},
"format": {
"anyOf": [
{
"$ref": "#/definitions/PageFormat"
},
{
"type": "null"
}
],
"description": "Page-reading mode: \"markdown\" | \"text\" | \"html\". When set, a browser UA + format Accept header are used, HTML is transformed, and response_format is ignored (treated as \"text\")."
},
"headers": {
"additionalProperties": {
"type": "string"
},
"default": null,
"description": "Request headers. Lower-cased keys recommended.",
"type": [
"object",
"null"
]
},
"include_links": {
"default": null,
"description": "Add `links: { internal, external }` to the envelope.",
"type": [
"boolean",
"null"
]
},
"include_media": {
"default": null,
"description": "Add `media: { images, videos, audios }` to the envelope.",
"type": [
"boolean",
"null"
]
},
"json": {
"default": null,
"description": "Structured JSON payload. When set, the handler stringifies it and forces `content-type: application/json`; wins over `body`."
},
"max_bytes": {
"default": null,
"description": "Cap on response body bytes. Defaults to the worker ceiling for raw fetches, or 256 KiB in page-reading mode (`format` set).",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"method": {
"default": null,
"description": "HTTP method. Defaults to GET. Case-insensitive (\"get\" works).",
"type": [
"string",
"null"
]
},
"response_format": {
"anyOf": [
{
"$ref": "#/definitions/ResponseFormat"
},
{
"type": "null"
}
],
"description": "How to return the body: \"text\" (default), \"base64\", or \"json\"."
},
"target_elements": {
"default": null,
"description": "CSS selectors (tl subset: tag/.class/#id). Restrict rendered content to these regions. Unmatched selectors are ignored.",
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"timeout_ms": {
"default": null,
"description": "Per-request timeout in ms. Capped by the worker's max_timeout_ms.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"url": {
"description": "Absolute http(s):// URL to fetch.",
"type": "string"
}
},
"required": [
"url"
],
"title": "FetchPayload",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Typed schema for the `web::fetch` response envelope. The handler returns a raw `Value` (its shape is request-dependent), so this type exists only to publish a concrete response schema: the SDK would otherwise emit the permissive `AnyValue` schema, which the registry rejects. Every field is optional because no single response carries all of them — success carries `ok`/`status`/`headers`/`body`/…, the page-mode viewable-image variant carries `content`+`details`, and failures carry `ok=false`/`error`/`message`.",
"properties": {
"body": {
"type": [
"string",
"null"
]
},
"bytes_truncated": {
"type": [
"boolean",
"null"
]
},
"content": true,
"content_type": {
"type": [
"string",
"null"
]
},
"details": true,
"error": {
"type": [
"string",
"null"
]
},
"headers": {
"additionalProperties": {
"type": "string"
},
"type": [
"object",
"null"
]
},
"json": true,
"links": true,
"media": true,
"message": {
"type": [
"string",
"null"
]
},
"ok": {
"type": [
"boolean",
"null"
]
},
"parse_error": {
"type": [
"string",
"null"
]
},
"redirect_chain": {
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"response_format": {
"type": [
"string",
"null"
]
},
"status": {
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"status_text": {
"type": [
"string",
"null"
]
},
"transformed": {
"type": [
"string",
"null"
]
}
},
"title": "FetchResponse",
"type": "object"
}
},
{
"description": "Internal pre_generate hook: appends web::fetch usage guidance to the agent system prompt. Bound to harness::hook::pre-generate at worker startup; not called directly.",
"metadata": {
"internal": true
},
"name": "web::inject-guidance",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"GenerateContext": {
"properties": {
"system_prompt": {
"default": "",
"description": "The system prompt assembled so far (base + any prior hook's mutation).",
"type": "string"
}
},
"type": "object"
}
},
"description": "The slice of the `pre_generate` hook envelope we read (lenient: ignores every other field the harness sends). The harness nests the live generation context under `generate` (see harness `HookRunner::run_pre_generate`).",
"properties": {
"generate": {
"$ref": "#/definitions/GenerateContext"
}
},
"title": "PreGenerateEvent",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"PreGenerateMutations": {
"description": "The harness applies `system_prompt` only when the key is present (`HookRunner::run_pre_generate` — `if m.system_prompt.is_some()`), so `None` serializes to an empty object: the safe no-op that preserves the harness's assembled prompt.",
"properties": {
"system_prompt": {
"description": "Full replacement system prompt (base + appended guidance). The harness overwrites, it does not merge.",
"type": [
"string",
"null"
]
}
},
"type": "object"
}
},
"description": "Hook envelope returned to the harness: the mutations to apply to the generation.",
"properties": {
"mutations": {
"$ref": "#/definitions/PreGenerateMutations"
}
},
"required": [
"mutations"
],
"title": "PreGenerateResponse",
"type": "object"
}
},
{
"description": "Internal: reload web settings from the authoritative configuration on change.",
"metadata": {
"internal": true
},
"name": "web::on-config-change",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "OnConfigChangeRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"ok": {
"type": "boolean"
}
},
"required": [
"ok"
],
"title": "OnConfigChangeResponse",
"type": "object"
}
}
],
"triggers": []
}