$worker

web

v1.2.3

Outbound HTTP client on the iii bus (web::fetch).

  • macOS: arm64 · x64
  • Linux: arm64 · armv7 · x64
  • Windows: arm64 · x64 · x86
agent-ready brief for v1.2.3
install + config + dependencies + readme + api reference, all in one place. fetch as agent-context.md for an llm to consume.
the same content rendered as discrete blocks below is exposed as a single markdown document at /workers/web.md. paste it into an llm prompt or pipe it through curl from a worker.

install

install
$iii worker add web@1.2.3

dependencies

dependencies1

readme

README.md

web

Outbound HTTP(S) client on the iii engine bus. A single bus function, web::fetch, performs a network request and returns a parsed { ok, status, headers, body } envelope with enforced size/timeout caps and server-side SSRF protection. It is the network counterpart to the shell worker: use web::fetch instead of shell::exec with curl/wget for any HTTP request.

The worker is a faithful Rust port of the original TypeScript web::fetch implementation — same request fields and success/error/image envelopes, so existing callers and the harness consumer are unaffected.

While connected it also injects a usage section into the agent system prompt via the harness pre-generate hook (web::inject-guidance), so the guidance is presence-gated: no web worker, no prompt text. The binding is one-shot at startup and relies on the engine's recoverable triggers (iii #1962, engine ≥ 0.21.8): bound before the harness is up, it parks as a pending intent and activates when the harness registers the trigger type. On older engines the bind is silently dropped.

Table of contents

  1. Install
  2. Configuration
  3. The web::fetch function
  4. Reading the result
  5. Page-reading mode
  6. SSRF guard
  7. Local development & testing

Install

iii worker add web

iii worker add fetches the binary (web), writes a config block into ~/.iii/config.yaml, and the engine starts the worker on the next iii start.


Configuration

The worker reads its operational ceilings from the configuration worker under the web key. It registers a schema + seed on boot, fetches the authoritative value, and hot-reloads on change. All keys are optional and fall back to the defaults below.

web:
  default_timeout_ms: 30000      # per-request timeout when caller omits timeout_ms
  max_timeout_ms: 120000         # ceiling; caller timeout_ms is clamped DOWN to this
  default_response_bytes: 262144 # default body cap in page-reading mode (256 KiB)
  max_response_bytes: 5242880    # absolute body ceiling (5 MiB); caller max_bytes clamped to this
  max_transform_bytes: 1048576   # HTML→markdown/text runs only on bodies ≤ this (1 MiB)
  max_redirects: 5               # hops before too_many_redirects
  user_agent: 'iii-harness/0.1 (+web::fetch)'
  allow_loopback: true           # set false in prod so 127.0.0.0/8, ::1, localhost are blocked

A copy of these defaults lives in config.yaml.example.


The web::fetch function

Minimal call — a URL is the only required field; everything else has a default:

// agent_trigger { function: "web::fetch", payload: { "url": "https://api.github.com/zen" } }
// → { "ok": true, "status": 200, "body": "Design for failure.", ... }

Request fields

Field Default Notes
url (required) absolute http(s)://
method GET case-insensitive; GET HEAD POST PUT PATCH DELETE OPTIONS
headers {} object; keys as written
json structured payload; auto-stringified + sets content-type: application/json. Wins over body.
body raw string body; ignored on GET/HEAD
response_format "text" "text" | "base64" (binary) | "json" (also parses into json)
format page-reading mode: "markdown" | "text" | "html" (see below)
content_filter { type: "pruning"|"bm25", query?, threshold?, threshold_type?, min_word_threshold? }. Page mode only. When set, body holds the filtered content (pruning default threshold 0.48; bm25 default 1.0; query falls back to page metadata).
target_elements CSS selectors (tl subset: tag/.class/#id); restrict rendered content to these regions
excluded_tags tag/selectors to drop before rendering (e.g. nav, footer)
include_links false adds links: { internal, external } (absolute URLs, classified by host)
include_media false adds media: { images, videos, audios } (absolute URLs)
timeout_ms default_timeout_ms clamped DOWN to max_timeout_ms
max_bytes max_response_bytes (default_response_bytes in format mode) over-cap body is truncated, not errored
follow_redirects true each hop re-checked against the SSRF blocklist

Response

Success (ok: true) — returned for any completed response, 2xx through 5xx:

{
  "ok": true,
  "status": 200,
  "status_text": "OK",
  "headers": { "content-type": "application/json", ... },  // keys LOWER-CASED
  "body": "<utf8 text | base64 when response_format=base64>",
  "json": { ... },            // only when response_format="json" and parse succeeded
  "parse_error": "…",         // only when response_format="json" and JSON.parse failed
  "response_format": "json",
  "bytes_truncated": false,   // true when body hit max_bytes (NOT an error)
  "redirect_chain": ["https://…/a"],  // omitted when no redirects
  "content_type": "text/html",  // page-reading mode only
  "transformed": "markdown",    // only when an HTML transform actually ran
  "links": { "internal": [{ "href": "…", "text": "…" }], "external": [ … ] },  // include_links
  "media": { "images": [{ "src": "…", "alt": "…" }], "videos": [ … ], "audios": [ … ] },  // include_media
}

Failure (ok: false) — the fetch never produced a response (no status):

{ "ok": false, "error": "blocked_host", "message": "…" }
error Cause Fix
invalid_payload Payload failed schema (bad method, wrong types) Correct the named fields
invalid_url url isn't a parseable absolute http(s):// URL Pass a full absolute URL incl. scheme
blocked_host Target resolves to a private / link-local / cloud-metadata IP Don't target internal/metadata hosts; for dev loopback set allow_loopback
timeout Slower than timeout_ms Raise timeout_ms (up to ceiling) or shrink work via max_bytes
too_many_redirects More than max_redirects hops Use the final URL, or follow_redirects: false and read the location header
transport_error Connection refused/reset, TLS/DNS failure, or unparseable redirect Location Check host/port/cert; retry if transient

There is no too_large error — oversize responses come back ok: true with bytes_truncated: true.


Reading the result

ok tells you if the request COMPLETED, not whether the server was happy.

  • A 404 or 500 is a successful fetchok: true, status: 404. Branch on status for HTTP outcomes.
  • ok: false means the request never produced a response. Branch on error.
if (!r.ok)              → fetch failed; look at r.error
else if (r.status>=400) → server returned an HTTP error; look at r.status / r.body
else                    → success; use r.body or r.json

Page-reading mode

response_format (transport encoding) and format (page-reading) are orthogonal — set one, not both. When format is set, response_format is ignored.

  • format: "markdown"text/html → Markdown (the right default for reading pages; far fewer tokens than raw HTML).
  • format: "text"text/html → plain text.
  • format: "html" — raw HTML.

When content_filter is set, body is the filtered output (there is no separate field) — feed it straight to a model. (If filtering would empty the page, or the page is too large or deeply nested to transform, body falls back to the unfiltered content.)

In page-reading mode the request goes out with a browser User-Agent + format-matched Accept header, and retries once with the honest UA on a Cloudflare cf-mitigated: challenge response. The HTML→markdown/text transform runs only on bodies ≤ max_transform_bytes; larger pages come back raw with transformed unset.

Images: a viewable image/* response (jpeg/png/gif/webp, 2xx, complete, non-empty) returns the actual image plus a one-line text summary instead of the JSON envelope. Anything else falls back to the normal envelope with response_format: "base64".


SSRF guard

DNS is resolved once, every resolved address is checked, then the request is dialed at the validated IP (pinned lookup + TLS servername) — so the IP that passed the check is the IP connected to (no DNS-rebind window). On each 3xx the Location is re-resolved and re-validated before following, and Authorization / Cookie / Proxy-Authorization are stripped when the redirect changes host or downgrades https → http.

Blocked unconditionallyerror: "blocked_host":

  • Private RFC1918 (10/8, 172.16/12, 192.168/16)
  • Link-local incl. cloud metadata 169.254.169.254
  • IPv6 ULA (fc00::/7), link-local (fe80::/10), multicast
  • ::ffff:-mapped IPv4 in both dotted (::ffff:169.254.169.254) and hex (::ffff:a9fe:a9fe) forms

Loopback (127.0.0.0/8, ::1, localhost) is allowed by default for dev convenience; operators set web.allow_loopback: false for prod, after which loopback returns blocked_host.


Local development & testing

cargo build                  # build the web binary
cargo test                   # unit + wiremock integration + adversarial SSRF tests
cargo clippy -- -D warnings  # lints
cargo fmt --check            # formatting

The agent-facing authoring guide lives in skills/index.md (served at runtime via directory::skills::get). For the exact, authoritative field types, call engine::functions::info { function_id: "web::fetch" } — the live schema wins if it ever disagrees with this document.

api reference (json)

agent-api-reference.json
{
  "functions": [
    {
      "description": "Fetch a URL over HTTP(S) and return the response as a structured envelope. Use this INSTEAD of `shell::exec` with curl for any HTTP request — it returns {ok, status, headers, body} as JSON, enforces size/timeout caps, and blocks private / cloud-metadata / link-local addresses server-side (SSRF guard; loopback is allowed by default for harness dev workflows). To READ A WEB PAGE, set `format: \"markdown\"` — HTML is converted to Markdown and images come back viewable (image responses in that mode return the image itself plus a text line, not the {ok,...} envelope). For JSON: pass `json: {...}` (auto-stringifies + sets content-type) and `response_format: \"json\"` (auto-parses response into the `json` field). Method is case-insensitive. On failure returns `{ok:false, error, message}` where `error` is one of: invalid_payload, invalid_url, blocked_host, timeout, too_many_redirects, transport_error. Branch on `error`, not on text.",
      "metadata": {},
      "name": "web::fetch",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "ContentFilter": {
            "description": "Content-filter request block. `query`, `threshold`, `threshold_type`, and `min_word_threshold` are optional; their default values are applied when the filter runs (see the content pipeline), not at deserialization.",
            "properties": {
              "min_word_threshold": {
                "default": null,
                "format": "uint32",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "query": {
                "default": null,
                "type": [
                  "string",
                  "null"
                ]
              },
              "threshold": {
                "default": null,
                "format": "double",
                "type": [
                  "number",
                  "null"
                ]
              },
              "threshold_type": {
                "anyOf": [
                  {
                    "$ref": "#/definitions/ThresholdType"
                  },
                  {
                    "type": "null"
                  }
                ]
              },
              "type": {
                "$ref": "#/definitions/FilterType"
              }
            },
            "required": [
              "type"
            ],
            "type": "object"
          },
          "FilterType": {
            "enum": [
              "pruning",
              "bm25"
            ],
            "type": "string"
          },
          "PageFormat": {
            "enum": [
              "markdown",
              "text",
              "html"
            ],
            "type": "string"
          },
          "ResponseFormat": {
            "enum": [
              "text",
              "base64",
              "json"
            ],
            "type": "string"
          },
          "ThresholdType": {
            "enum": [
              "fixed",
              "dynamic"
            ],
            "type": "string"
          }
        },
        "properties": {
          "body": {
            "default": null,
            "description": "Raw request body as a string. Prefer `json` for JSON payloads.",
            "type": [
              "string",
              "null"
            ]
          },
          "content_filter": {
            "anyOf": [
              {
                "$ref": "#/definitions/ContentFilter"
              },
              {
                "type": "null"
              }
            ],
            "description": "Content filter: prunes boilerplate. When set, `body` holds the filtered output in the requested `format` (page-reading mode only)."
          },
          "excluded_tags": {
            "default": null,
            "description": "Tag names / selectors to drop before rendering (e.g. \"nav\", \"footer\").",
            "items": {
              "type": "string"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "follow_redirects": {
            "default": null,
            "description": "Follow 3xx redirects. Defaults to true. Each hop is re-checked against the SSRF blocklist.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "format": {
            "anyOf": [
              {
                "$ref": "#/definitions/PageFormat"
              },
              {
                "type": "null"
              }
            ],
            "description": "Page-reading mode: \"markdown\" | \"text\" | \"html\". When set, a browser UA + format Accept header are used, HTML is transformed, and response_format is ignored (treated as \"text\")."
          },
          "headers": {
            "additionalProperties": {
              "type": "string"
            },
            "default": null,
            "description": "Request headers. Lower-cased keys recommended.",
            "type": [
              "object",
              "null"
            ]
          },
          "include_links": {
            "default": null,
            "description": "Add `links: { internal, external }` to the envelope.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "include_media": {
            "default": null,
            "description": "Add `media: { images, videos, audios }` to the envelope.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "json": {
            "default": null,
            "description": "Structured JSON payload. When set, the handler stringifies it and forces `content-type: application/json`; wins over `body`."
          },
          "max_bytes": {
            "default": null,
            "description": "Cap on response body bytes. Defaults to the worker ceiling for raw fetches, or 256 KiB in page-reading mode (`format` set).",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "method": {
            "default": null,
            "description": "HTTP method. Defaults to GET. Case-insensitive (\"get\" works).",
            "type": [
              "string",
              "null"
            ]
          },
          "response_format": {
            "anyOf": [
              {
                "$ref": "#/definitions/ResponseFormat"
              },
              {
                "type": "null"
              }
            ],
            "description": "How to return the body: \"text\" (default), \"base64\", or \"json\"."
          },
          "target_elements": {
            "default": null,
            "description": "CSS selectors (tl subset: tag/.class/#id). Restrict rendered content to these regions. Unmatched selectors are ignored.",
            "items": {
              "type": "string"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "timeout_ms": {
            "default": null,
            "description": "Per-request timeout in ms. Capped by the worker's max_timeout_ms.",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "url": {
            "description": "Absolute http(s):// URL to fetch.",
            "type": "string"
          }
        },
        "required": [
          "url"
        ],
        "title": "FetchPayload",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Typed schema for the `web::fetch` response envelope. The handler returns a raw `Value` (its shape is request-dependent), so this type exists only to publish a concrete response schema: the SDK would otherwise emit the permissive `AnyValue` schema, which the registry rejects. Every field is optional because no single response carries all of them — success carries `ok`/`status`/`headers`/`body`/…, the page-mode viewable-image variant carries `content`+`details`, and failures carry `ok=false`/`error`/`message`.",
        "properties": {
          "body": {
            "type": [
              "string",
              "null"
            ]
          },
          "bytes_truncated": {
            "type": [
              "boolean",
              "null"
            ]
          },
          "content": true,
          "content_type": {
            "type": [
              "string",
              "null"
            ]
          },
          "details": true,
          "error": {
            "type": [
              "string",
              "null"
            ]
          },
          "headers": {
            "additionalProperties": {
              "type": "string"
            },
            "type": [
              "object",
              "null"
            ]
          },
          "json": true,
          "links": true,
          "media": true,
          "message": {
            "type": [
              "string",
              "null"
            ]
          },
          "ok": {
            "type": [
              "boolean",
              "null"
            ]
          },
          "parse_error": {
            "type": [
              "string",
              "null"
            ]
          },
          "redirect_chain": {
            "items": {
              "type": "string"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "response_format": {
            "type": [
              "string",
              "null"
            ]
          },
          "status": {
            "format": "uint16",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "status_text": {
            "type": [
              "string",
              "null"
            ]
          },
          "transformed": {
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "FetchResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal pre_generate hook: appends web::fetch usage guidance to the agent system prompt. Bound to harness::hook::pre-generate at worker startup; not called directly.",
      "metadata": {
        "internal": true
      },
      "name": "web::inject-guidance",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "GenerateContext": {
            "properties": {
              "system_prompt": {
                "default": "",
                "description": "The system prompt assembled so far (base + any prior hook's mutation).",
                "type": "string"
              }
            },
            "type": "object"
          }
        },
        "description": "The slice of the `pre_generate` hook envelope we read (lenient: ignores every other field the harness sends). The harness nests the live generation context under `generate` (see harness `HookRunner::run_pre_generate`).",
        "properties": {
          "generate": {
            "$ref": "#/definitions/GenerateContext"
          }
        },
        "title": "PreGenerateEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "PreGenerateMutations": {
            "description": "The harness applies `system_prompt` only when the key is present (`HookRunner::run_pre_generate` — `if m.system_prompt.is_some()`), so `None` serializes to an empty object: the safe no-op that preserves the harness's assembled prompt.",
            "properties": {
              "system_prompt": {
                "description": "Full replacement system prompt (base + appended guidance). The harness overwrites, it does not merge.",
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "type": "object"
          }
        },
        "description": "Hook envelope returned to the harness: the mutations to apply to the generation.",
        "properties": {
          "mutations": {
            "$ref": "#/definitions/PreGenerateMutations"
          }
        },
        "required": [
          "mutations"
        ],
        "title": "PreGenerateResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal: reload web settings from the authoritative configuration on change.",
      "metadata": {
        "internal": true
      },
      "name": "web::on-config-change",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "OnConfigChangeRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "ok": {
            "type": "boolean"
          }
        },
        "required": [
          "ok"
        ],
        "title": "OnConfigChangeResponse",
        "type": "object"
      }
    }
  ],
  "triggers": []
}