# web

> Outbound HTTP client on the iii bus (web::fetch).

| field | value |
|-------|-------|
| version | 1.1.4 |
| type | binary |
| repo | https://github.com/iii-hq/workers |
| supported_targets | x86_64-apple-darwin, aarch64-apple-darwin, i686-pc-windows-msvc, x86_64-pc-windows-msvc, aarch64-pc-windows-msvc, x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-unknown-linux-musl, armv7-unknown-linux-gnueabihf |
| author | iii |

## installation

```sh
iii worker add web@1.1.4
```

## dependencies

- `configuration` @ `^0.19.0`

## readme

# web

Outbound HTTP(S) client on the [iii engine](https://github.com/iii-hq/iii)
bus. A single bus function, `web::fetch`, performs a network request and
returns a parsed `{ ok, status, headers, body }` envelope with enforced
size/timeout caps and server-side SSRF protection. It is the network
counterpart to the `shell` worker: use `web::fetch` instead of
`shell::exec` with `curl`/`wget` for any HTTP request.

The worker is a faithful Rust port of the original TypeScript `web::fetch`
implementation — same request fields and success/error/image envelopes, so
existing callers and the harness consumer are unaffected.

## Table of contents

1. [Install](#install)
2. [Configuration](#configuration)
3. [The `web::fetch` function](#the-webfetch-function)
4. [Reading the result](#reading-the-result)
5. [Page-reading mode](#page-reading-mode)
6. [SSRF guard](#ssrf-guard)
7. [Local development & testing](#local-development--testing)

---

## Install

```bash
iii worker add web
```

`iii worker add` fetches the binary (`web`), writes a config block into
`~/.iii/config.yaml`, and the engine starts the worker on the next
`iii start`.

---

## Configuration

The worker reads its operational ceilings from the `configuration` worker
under the `web` key. It registers a schema + seed on boot, fetches the
authoritative value, and hot-reloads on change. All keys are optional and
fall back to the defaults below.

```yaml
web:
  default_timeout_ms: 30000      # per-request timeout when caller omits timeout_ms
  max_timeout_ms: 120000         # ceiling; caller timeout_ms is clamped DOWN to this
  default_response_bytes: 262144 # default body cap in page-reading mode (256 KiB)
  max_response_bytes: 5242880    # absolute body ceiling (5 MiB); caller max_bytes clamped to this
  max_transform_bytes: 1048576   # HTML→markdown/text runs only on bodies ≤ this (1 MiB)
  max_redirects: 5               # hops before too_many_redirects
  user_agent: 'iii-harness/0.1 (+web::fetch)'
  allow_loopback: true           # set false in prod so 127.0.0.0/8, ::1, localhost are blocked
```

A copy of these defaults lives in [`config.yaml.example`](./config.yaml.example).

---

## The `web::fetch` function

Minimal call — a URL is the only required field; everything else has a
default:

```jsonc
// agent_trigger { function: "web::fetch", payload: { "url": "https://api.github.com/zen" } }
// → { "ok": true, "status": 200, "body": "Design for failure.", ... }
```

### Request fields

| Field | Default | Notes |
|---|---|---|
| `url` *(required)* | — | absolute `http(s)://` |
| `method` | `GET` | case-insensitive; GET HEAD POST PUT PATCH DELETE OPTIONS |
| `headers` | `{}` | object; keys as written |
| `json` | — | structured payload; auto-stringified + sets `content-type: application/json`. **Wins over `body`.** |
| `body` | — | raw string body; ignored on GET/HEAD |
| `response_format` | `"text"` | `"text"` \| `"base64"` (binary) \| `"json"` (also parses into `json`) |
| `format` | — | page-reading mode: `"markdown"` \| `"text"` \| `"html"` (see below) |
| `timeout_ms` | `default_timeout_ms` | clamped DOWN to `max_timeout_ms` |
| `max_bytes` | `max_response_bytes` (`default_response_bytes` in `format` mode) | over-cap body is truncated, not errored |
| `follow_redirects` | `true` | each hop re-checked against the SSRF blocklist |

### Response

**Success** (`ok: true`) — returned for *any* completed response, 2xx through 5xx:

```jsonc
{
  "ok": true,
  "status": 200,
  "status_text": "OK",
  "headers": { "content-type": "application/json", ... },  // keys LOWER-CASED
  "body": "<utf8 text | base64 when response_format=base64>",
  "json": { ... },            // only when response_format="json" and parse succeeded
  "parse_error": "…",         // only when response_format="json" and JSON.parse failed
  "response_format": "json",
  "bytes_truncated": false,   // true when body hit max_bytes (NOT an error)
  "redirect_chain": ["https://…/a"],  // omitted when no redirects
  "content_type": "text/html",  // page-reading mode only
  "transformed": "markdown"     // only when an HTML transform actually ran
}
```

**Failure** (`ok: false`) — the fetch never produced a response (no `status`):

```jsonc
{ "ok": false, "error": "blocked_host", "message": "…" }
```

| `error` | Cause | Fix |
|---|---|---|
| `invalid_payload` | Payload failed schema (bad `method`, wrong types) | Correct the named fields |
| `invalid_url` | `url` isn't a parseable absolute `http(s)://` URL | Pass a full absolute URL incl. scheme |
| `blocked_host` | Target resolves to a private / link-local / cloud-metadata IP | Don't target internal/metadata hosts; for dev loopback set `allow_loopback` |
| `timeout` | Slower than `timeout_ms` | Raise `timeout_ms` (up to ceiling) or shrink work via `max_bytes` |
| `too_many_redirects` | More than `max_redirects` hops | Use the final URL, or `follow_redirects: false` and read the `location` header |
| `transport_error` | Connection refused/reset, TLS/DNS failure, or unparseable redirect `Location` | Check host/port/cert; retry if transient |

There is **no `too_large` error** — oversize responses come back `ok: true`
with `bytes_truncated: true`.

---

## Reading the result

**`ok` tells you if the request COMPLETED, not whether the server was happy.**

- A 404 or 500 is a **successful fetch** → `ok: true`, `status: 404`. Branch on `status` for HTTP outcomes.
- `ok: false` means the request never produced a response. Branch on `error`.

```jsonc
if (!r.ok)              → fetch failed; look at r.error
else if (r.status>=400) → server returned an HTTP error; look at r.status / r.body
else                    → success; use r.body or r.json
```

---

## Page-reading mode

`response_format` (transport encoding) and `format` (page-reading) are
orthogonal — set one, not both. When `format` is set, `response_format` is
ignored.

- **`format: "markdown"`** — `text/html` → Markdown (the right default for reading pages; far fewer tokens than raw HTML).
- **`format: "text"`** — `text/html` → plain text.
- **`format: "html"`** — raw HTML.

In page-reading mode the request goes out with a browser User-Agent +
format-matched `Accept` header, and retries once with the honest UA on a
Cloudflare `cf-mitigated: challenge` response. The HTML→markdown/text
transform runs only on bodies ≤ `max_transform_bytes`; larger pages come
back raw with `transformed` unset.

**Images:** a viewable `image/*` response (jpeg/png/gif/webp, 2xx,
complete, non-empty) returns the actual image plus a one-line text summary
instead of the JSON envelope. Anything else falls back to the normal
envelope with `response_format: "base64"`.

---

## SSRF guard

DNS is resolved **once**, every resolved address is checked, then the
request is dialed at the **validated IP** (pinned `lookup` + TLS
`servername`) — so the IP that passed the check is the IP connected to (no
DNS-rebind window). On each 3xx the `Location` is re-resolved and
re-validated before following, and `Authorization` / `Cookie` /
`Proxy-Authorization` are stripped when the redirect changes host or
downgrades `https → http`.

Blocked **unconditionally** → `error: "blocked_host"`:

- Private RFC1918 (`10/8`, `172.16/12`, `192.168/16`)
- Link-local incl. **cloud metadata** `169.254.169.254`
- IPv6 ULA (`fc00::/7`), link-local (`fe80::/10`), multicast
- `::ffff:`-mapped IPv4 in both dotted (`::ffff:169.254.169.254`) and hex (`::ffff:a9fe:a9fe`) forms

Loopback (`127.0.0.0/8`, `::1`, `localhost`) is **allowed by default** for
dev convenience; operators set `web.allow_loopback: false` for prod, after
which loopback returns `blocked_host`.

---

## Local development & testing

```bash
cargo build                  # build the web binary
cargo test                   # unit + wiremock integration + adversarial SSRF tests
cargo clippy -- -D warnings  # lints
cargo fmt --check            # formatting
```

The agent-facing authoring guide lives in
[`skills/index.md`](./skills/index.md) (served at runtime via
`directory::skills::get`). For the exact, authoritative field types, call
`engine::functions::info { function_id: "web::fetch" }` — the live schema
wins if it ever disagrees with this document.

## api reference

```json
{
  "functions": [
    {
      "description": "Fetch a URL over HTTP(S) and return the response as a structured envelope. Use this INSTEAD of `shell::exec` with curl for any HTTP request — it returns {ok, status, headers, body} as JSON, enforces size/timeout caps, and blocks private / cloud-metadata / link-local addresses server-side (SSRF guard; loopback is allowed by default for harness dev workflows). To READ A WEB PAGE, set `format: \"markdown\"` — HTML is converted to Markdown and images come back viewable (image responses in that mode return the image itself plus a text line, not the {ok,...} envelope). For JSON: pass `json: {...}` (auto-stringifies + sets content-type) and `response_format: \"json\"` (auto-parses response into the `json` field). Method is case-insensitive. On failure returns `{ok:false, error, message}` where `error` is one of: invalid_payload, invalid_url, blocked_host, timeout, too_many_redirects, transport_error. Branch on `error`, not on text.",
      "metadata": {},
      "name": "web::fetch",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "PageFormat": {
            "enum": [
              "markdown",
              "text",
              "html"
            ],
            "type": "string"
          },
          "ResponseFormat": {
            "enum": [
              "text",
              "base64",
              "json"
            ],
            "type": "string"
          }
        },
        "properties": {
          "body": {
            "default": null,
            "description": "Raw request body as a string. Prefer `json` for JSON payloads.",
            "type": [
              "string",
              "null"
            ]
          },
          "follow_redirects": {
            "default": null,
            "description": "Follow 3xx redirects. Defaults to true. Each hop is re-checked against the SSRF blocklist.",
            "type": [
              "boolean",
              "null"
            ]
          },
          "format": {
            "anyOf": [
              {
                "$ref": "#/definitions/PageFormat"
              },
              {
                "type": "null"
              }
            ],
            "description": "Page-reading mode: \"markdown\" | \"text\" | \"html\". When set, a browser UA + format Accept header are used, HTML is transformed, and response_format is ignored (treated as \"text\")."
          },
          "headers": {
            "additionalProperties": {
              "type": "string"
            },
            "default": null,
            "description": "Request headers. Lower-cased keys recommended.",
            "type": [
              "object",
              "null"
            ]
          },
          "json": {
            "default": null,
            "description": "Structured JSON payload. When set, the handler stringifies it and forces `content-type: application/json`; wins over `body`."
          },
          "max_bytes": {
            "default": null,
            "description": "Cap on response body bytes. Defaults to the worker ceiling for raw fetches, or 256 KiB in page-reading mode (`format` set).",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "method": {
            "default": null,
            "description": "HTTP method. Defaults to GET. Case-insensitive (\"get\" works).",
            "type": [
              "string",
              "null"
            ]
          },
          "response_format": {
            "anyOf": [
              {
                "$ref": "#/definitions/ResponseFormat"
              },
              {
                "type": "null"
              }
            ],
            "description": "How to return the body: \"text\" (default), \"base64\", or \"json\"."
          },
          "timeout_ms": {
            "default": null,
            "description": "Per-request timeout in ms. Capped by the worker's max_timeout_ms.",
            "format": "uint64",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "url": {
            "description": "Absolute http(s):// URL to fetch.",
            "type": "string"
          }
        },
        "required": [
          "url"
        ],
        "title": "FetchPayload",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Typed schema for the `web::fetch` response envelope. The handler returns a raw `Value` (its shape is request-dependent), so this type exists only to publish a concrete response schema: the SDK would otherwise emit the permissive `AnyValue` schema, which the registry rejects. Every field is optional because no single response carries all of them — success carries `ok`/`status`/`headers`/`body`/…, the page-mode viewable-image variant carries `content`+`details`, and failures carry `ok=false`/`error`/`message`.",
        "properties": {
          "body": {
            "type": [
              "string",
              "null"
            ]
          },
          "bytes_truncated": {
            "type": [
              "boolean",
              "null"
            ]
          },
          "content": true,
          "content_type": {
            "type": [
              "string",
              "null"
            ]
          },
          "details": true,
          "error": {
            "type": [
              "string",
              "null"
            ]
          },
          "headers": {
            "additionalProperties": {
              "type": "string"
            },
            "type": [
              "object",
              "null"
            ]
          },
          "json": true,
          "message": {
            "type": [
              "string",
              "null"
            ]
          },
          "ok": {
            "type": [
              "boolean",
              "null"
            ]
          },
          "parse_error": {
            "type": [
              "string",
              "null"
            ]
          },
          "redirect_chain": {
            "items": {
              "type": "string"
            },
            "type": [
              "array",
              "null"
            ]
          },
          "response_format": {
            "type": [
              "string",
              "null"
            ]
          },
          "status": {
            "format": "uint16",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "status_text": {
            "type": [
              "string",
              "null"
            ]
          },
          "transformed": {
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "FetchResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal: reload web settings from the authoritative configuration on change.",
      "metadata": {},
      "name": "web::on-config-change",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "OnConfigChangeRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "ok": {
            "type": "boolean"
          }
        },
        "required": [
          "ok"
        ],
        "title": "OnConfigChangeResponse",
        "type": "object"
      }
    }
  ],
  "triggers": []
}
```