# scrapling

> Scrapling as an iii worker — scrapling::* functions run HTTP / anti-bot / browser fetches, CSS/XPath/regex/adaptive extraction, persistent sessions, and a declarative crawl over the iii bus.

| field | value |
|-------|-------|
| version | 0.2.6 |
| type | bundle |
| repo | https://github.com/iii-hq/workers |
| author | iii |

## installation

```sh
iii worker add scrapling@0.2.6
```

## configuration

```yaml
- adaptive_storage_path: ./data/scrapling/elements.db
  defaults:
    headless: true
    impersonate: chrome
    include_html: false
    network_idle: false
    proxy: 
  engine_url: ws://127.0.0.1:49134
  max_bulk_concurrency: 5
  max_sessions: 8
  session_idle_timeout_s: 900
```

## readme

# scrapling

[Scrapling](https://github.com/D4Vinci/Scrapling) as an iii worker. It maps
Scrapling's three fetch tiers and its parsing engine to `scrapling::*` functions
on the iii bus: fast HTTP with TLS impersonation, a Camoufox anti-bot browser, a
full Playwright/Chromium browser, screenshots, and CSS/XPath/regex/adaptive
extraction.

While connected it also injects a usage section into the agent system prompt
via the harness `pre-generate` hook (`scrapling::inject-guidance`), so the
guidance is presence-gated: no scrapling worker, no prompt text. The binding is
one-shot at startup and relies on the engine's recoverable triggers (iii #1962,
engine ≥ 0.21.8): bound before the harness is up, it parks as a pending intent
and activates when the harness registers the trigger type. On older engines the
bind is silently dropped.

## Install

```bash
iii worker add scrapling
```

The worker is a `deploy: image` Python worker. The image build runs
`scrapling install`, which downloads the Camoufox and Chromium browsers used by
`stealthy-fetch`, `dynamic-fetch`, and `screenshot`.

## Functions

| Function | What it does |
|---|---|
| `scrapling::fetch` | HTTP get/post/put/delete (curl_cffi, TLS impersonation) |
| `scrapling::stealthy-fetch` | Camoufox stealth browser — Cloudflare/Turnstile bypass, WebRTC/canvas hardening |
| `scrapling::dynamic-fetch` | Playwright/Chromium — JS render, waits, XHR capture, CDP |
| `scrapling::screenshot` | Page screenshot (image content blocks) via a browser fetcher |
| `scrapling::extract` | Parse HTML with a declarative selector list |
| `scrapling::css` | One CSS query over provided HTML |
| `scrapling::xpath` | One XPath query over provided HTML |
| `scrapling::regex` | Regex over the visible text of provided HTML |
| `scrapling::find-similar` | An example element + structurally similar elements |

### Fetch output

Fetch functions return:

```json
{ "status": 200, "url": "...", "headers": {}, "cookies": {}, "encoding": "utf-8",
  "extracted": { "...": "..." }, "html": "<...>" }
```

`extracted` appears only when `selectors` are given; `html` only when
`include_html: true`. Called with a `urls` array, the response is
`{ "results": [ ...one object per url... ] }` (a failed URL yields
`{ "url", "error" }`, so one bad URL doesn't sink the batch).

### Selector contract

`selectors` (on the fetchers and `scrapling::extract`) is a list of specs:

```json
[
  { "name": "title", "css": "h1" },
  { "name": "links", "css": "a", "attr": "href", "all": true },
  { "name": "price", "regex": "price (\\d+)" },
  { "name": "card_html", "css": ".card", "html": true }
]
```

- one of `css` / `xpath` / `regex` per spec
- `attr` pulls an attribute; `html` pulls inner HTML; otherwise text
- `all: true` returns every match as a list, else the first match (or `null`)

## Examples

```bash
# HTTP fetch + extract in one call
iii trigger scrapling::fetch --payload '{
  "url": "https://example.com",
  "selectors": [{ "name": "title", "css": "title" }]
}'

# Anti-bot page
iii trigger scrapling::stealthy-fetch --payload '{
  "url": "https://nopecha.com/demo/cloudflare",
  "solve_cloudflare": true,
  "selectors": [{ "name": "body", "css": "#padded_content a", "all": true }]
}'

# Parse HTML you already have
iii trigger scrapling::extract --payload '{
  "html": "<ul><li>a</li><li>b</li></ul>",
  "selectors": [{ "name": "items", "css": "li", "all": true }]
}'

# Fetch many URLs at once
iii trigger scrapling::fetch --payload '{ "urls": ["https://a.com", "https://b.com"] }'
```

## Config

`config.yaml` holds operator defaults applied when a call omits the field:

```yaml
defaults:
  impersonate: chrome     # HTTP fetcher fingerprint
  headless: true          # browser fetchers
  network_idle: false
  proxy: ""               # "" = none
  include_html: false
max_bulk_concurrency: 5
```

`timeout` is passed per call, not defaulted here — it means **seconds** for the
HTTP fetcher and **milliseconds** for the browser fetchers.

## How it maps

| Scrapling | iii |
|---|---|
| `Fetcher.{get,post,put,delete}` | `scrapling::fetch` |
| `StealthyFetcher.fetch` | `scrapling::stealthy-fetch` |
| `DynamicFetcher.fetch` | `scrapling::dynamic-fetch` |
| browser `page.screenshot()` | `scrapling::screenshot` |
| `Selector.css / .xpath / .re / .find_similar` | `extract` / `css` / `xpath` / `regex` / `find-similar` |

## Boundaries

- Non-JSON Scrapling options (Python `page_action`/`page_setup` callbacks, proxy
  rotators, persistent sessions, the `Spider` crawl layer) are not exposed. Pass
  a single `proxy` string.
- The fetch functions are not agent-callable without human approval (outbound
  requests to arbitrary URLs); the pure parsers are. See `iii-permissions.yaml`.

## api reference

```json
{
  "functions": [
    {
      "description": "BFS-crawl from start_urls (follow same-domain links), extract per page, stream items.",
      "metadata": {},
      "name": "scrapling::crawl",
      "request_schema": {
        "properties": {
          "allowed_domains": {
            "description": "only follow links on these hosts",
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "concurrency": {
            "type": "integer"
          },
          "css_selector": {
            "description": "scope the render to this CSS subtree (e.g. a page's content div)",
            "type": "string"
          },
          "download_delay": {
            "description": "seconds to wait between crawl rounds",
            "type": "number"
          },
          "fetcher": {
            "enum": [
              "http",
              "stealthy",
              "dynamic"
            ],
            "type": "string"
          },
          "format": {
            "description": "render page body to this format",
            "enum": [
              "markdown",
              "text"
            ],
            "type": "string"
          },
          "impersonate": {
            "type": "string"
          },
          "include_html": {
            "type": "boolean"
          },
          "main_content_only": {
            "description": "strip nav/scripts/hidden before rendering",
            "type": "boolean"
          },
          "max_depth": {
            "type": "integer"
          },
          "max_pages": {
            "type": "integer"
          },
          "same_domain": {
            "description": "follow only same-host links (default true)",
            "type": "boolean"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "start_urls": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "stream_name": {
            "description": "stream to emit items on (default scrapling::crawl)",
            "type": "string"
          },
          "url": {
            "description": "single start URL (alternative to start_urls)",
            "type": "string"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "items": {
            "description": "a small sample of streamed items",
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "stats": {
            "properties": {
              "crawled": {
                "type": "integer"
              },
              "errors": {
                "type": "integer"
              },
              "items": {
                "type": "integer"
              },
              "stopped": {
                "type": "string"
              }
            },
            "type": "object"
          },
          "stream": {
            "description": "read the full item stream via stream::on with this name + group_id",
            "properties": {
              "group_id": {
                "type": "string"
              },
              "name": {
                "type": "string"
              }
            },
            "type": "object"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "One CSS query over HTML; first-or-all; `attr` pulls an attribute else text.",
      "metadata": {},
      "name": "scrapling::css",
      "request_schema": {
        "properties": {
          "adaptive": {
            "description": "relocate elements after a site change via saved identities",
            "type": "boolean"
          },
          "adaptive_domain": {
            "description": "page URL/domain that keys saved identities",
            "type": "string"
          },
          "attr": {
            "type": "string"
          },
          "auto_save": {
            "description": "save matched identities (defaults on when adaptive)",
            "type": "boolean"
          },
          "first": {
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "identifier": {
            "description": "stable key for the saved element",
            "type": "string"
          },
          "query": {
            "type": "string"
          }
        },
        "required": [
          "html",
          "query"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "result": {
            "items": {
              "type": [
                "string",
                "null"
              ]
            },
            "type": [
              "array",
              "string",
              "null"
            ]
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Describe the first css/xpath match: attrs, generated selectors, class list, DOM context.",
      "metadata": {},
      "name": "scrapling::describe",
      "request_schema": {
        "properties": {
          "html": {
            "type": "string"
          },
          "kind": {
            "enum": [
              "css",
              "xpath"
            ],
            "type": "string"
          },
          "query": {
            "type": "string"
          }
        },
        "required": [
          "html",
          "query"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "element": {
            "properties": {
              "attrs": {
                "type": "object"
              },
              "children": {
                "type": "integer"
              },
              "classes": {
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "css": {
                "type": "string"
              },
              "full_css": {
                "type": "string"
              },
              "full_xpath": {
                "type": "string"
              },
              "html": {
                "type": "string"
              },
              "parent_tag": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "siblings": {
                "type": "integer"
              },
              "tag": {
                "type": "string"
              },
              "text": {
                "type": "string"
              },
              "xpath": {
                "type": "string"
              }
            },
            "type": "object"
          },
          "found": {
            "type": "boolean"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Playwright/Chromium fetch: JS render, waits, XHR capture, CDP; extraction + bulk.",
      "metadata": {},
      "name": "scrapling::dynamic-fetch",
      "request_schema": {
        "properties": {
          "block_ads": {
            "type": "boolean"
          },
          "blocked_domains": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "capture_xhr": {
            "type": "string"
          },
          "cdp_url": {
            "type": "string"
          },
          "cookies": {
            "type": "object"
          },
          "css_selector": {
            "description": "scope the render to this CSS subtree (e.g. a page's content div)",
            "type": "string"
          },
          "disable_resources": {
            "type": "boolean"
          },
          "dns_over_https": {
            "type": "boolean"
          },
          "extra_flags": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "extra_headers": {
            "type": "object"
          },
          "format": {
            "description": "render page body to this format",
            "enum": [
              "markdown",
              "text"
            ],
            "type": "string"
          },
          "google_search": {
            "type": "boolean"
          },
          "headless": {
            "type": "boolean"
          },
          "include_html": {
            "type": "boolean"
          },
          "load_dom": {
            "type": "boolean"
          },
          "locale": {
            "type": "string"
          },
          "main_content_only": {
            "description": "strip nav/scripts/hidden before rendering",
            "type": "boolean"
          },
          "max_pages": {
            "type": "integer"
          },
          "network_idle": {
            "type": "boolean"
          },
          "proxy": {
            "type": "string"
          },
          "real_chrome": {
            "type": "boolean"
          },
          "retries": {
            "type": "integer"
          },
          "retry_delay": {
            "type": "number"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "timeout": {
            "description": "milliseconds (browser fetcher)",
            "type": "number"
          },
          "timezone_id": {
            "type": "string"
          },
          "url": {
            "type": "string"
          },
          "urls": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "useragent": {
            "type": "string"
          },
          "wait": {
            "description": "extra ms to wait after load",
            "type": "number"
          },
          "wait_selector": {
            "type": "string"
          },
          "wait_selector_state": {
            "enum": [
              "attached",
              "detached",
              "visible",
              "hidden"
            ],
            "type": "string"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "captured_xhr": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "content": {
            "description": "markdown/text render when `format` requested",
            "type": "string"
          },
          "cookies": {
            "type": "object"
          },
          "encoding": {
            "type": [
              "string",
              "null"
            ]
          },
          "error": {
            "type": "string"
          },
          "extracted": {
            "type": "object"
          },
          "format": {
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "html": {
            "type": "string"
          },
          "results": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "status": {
            "type": [
              "integer",
              "null"
            ]
          },
          "url": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Parse HTML with a selector list (css/xpath/regex, text/attr/html, all-or-first).",
      "metadata": {},
      "name": "scrapling::extract",
      "request_schema": {
        "properties": {
          "adaptive": {
            "description": "relocate elements after a site change via saved identities",
            "type": "boolean"
          },
          "adaptive_domain": {
            "description": "page URL/domain that keys saved identities",
            "type": "string"
          },
          "auto_save": {
            "description": "save matched identities (defaults on when adaptive)",
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          }
        },
        "required": [
          "html",
          "selectors"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "extracted": {
            "type": "object"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Fast HTTP fetch, TLS impersonation: get/post/put/delete, inline extraction, bulk `urls`.",
      "metadata": {},
      "name": "scrapling::fetch",
      "request_schema": {
        "properties": {
          "cookies": {
            "type": "object"
          },
          "css_selector": {
            "description": "scope the render to this CSS subtree (e.g. a page's content div)",
            "type": "string"
          },
          "data": {
            "type": "object"
          },
          "follow_redirects": {
            "type": "boolean"
          },
          "format": {
            "description": "render page body to this format",
            "enum": [
              "markdown",
              "text"
            ],
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "http3": {
            "type": "boolean"
          },
          "impersonate": {
            "description": "TLS/UA fingerprint, e.g. 'chrome'",
            "type": "string"
          },
          "include_html": {
            "type": "boolean"
          },
          "json": {
            "type": "object"
          },
          "main_content_only": {
            "description": "strip nav/scripts/hidden before rendering",
            "type": "boolean"
          },
          "max_redirects": {
            "type": "integer"
          },
          "method": {
            "enum": [
              "get",
              "post",
              "put",
              "delete"
            ],
            "type": "string"
          },
          "params": {
            "type": "object"
          },
          "proxies": {
            "description": "per-scheme proxies, e.g. {\"https\": \"http://...\"}",
            "type": "object"
          },
          "proxy": {
            "type": "string"
          },
          "proxy_auth": {
            "description": "[user, password]",
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "retries": {
            "type": "integer"
          },
          "retry_delay": {
            "type": "number"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "stealthy_headers": {
            "type": "boolean"
          },
          "timeout": {
            "description": "seconds (HTTP fetcher)",
            "type": "number"
          },
          "url": {
            "type": "string"
          },
          "urls": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "verify": {
            "type": "boolean"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "captured_xhr": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "content": {
            "description": "markdown/text render when `format` requested",
            "type": "string"
          },
          "cookies": {
            "type": "object"
          },
          "encoding": {
            "type": [
              "string",
              "null"
            ]
          },
          "error": {
            "type": "string"
          },
          "extracted": {
            "type": "object"
          },
          "format": {
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "html": {
            "type": "string"
          },
          "results": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "status": {
            "type": [
              "integer",
              "null"
            ]
          },
          "url": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Find elements by tag/attribute filters (+ optional text regex); BeautifulSoup-style.",
      "metadata": {},
      "name": "scrapling::find",
      "request_schema": {
        "properties": {
          "attrs": {
            "description": "attribute filters, e.g. {\"class\": \"card\"}",
            "type": "object"
          },
          "first": {
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "limit": {
            "type": "integer"
          },
          "tag": {
            "description": "tag name or list of tag names",
            "items": {
              "type": "string"
            },
            "type": [
              "string",
              "array"
            ]
          },
          "text_regex": {
            "description": "keep only elements whose text matches this regex",
            "type": "string"
          }
        },
        "required": [
          "html"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "count": {
            "type": "integer"
          },
          "items": {
            "items": {
              "properties": {
                "attrs": {
                  "type": "object"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "type": "string"
                },
                "tag": {
                  "type": "string"
                },
                "text": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Find elements whose visible text matches a regex pattern.",
      "metadata": {},
      "name": "scrapling::find-by-regex",
      "request_schema": {
        "properties": {
          "case_sensitive": {
            "type": "boolean"
          },
          "clean_match": {
            "type": "boolean"
          },
          "first": {
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "limit": {
            "type": "integer"
          },
          "pattern": {
            "type": "string"
          }
        },
        "required": [
          "html",
          "pattern"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "count": {
            "type": "integer"
          },
          "items": {
            "items": {
              "properties": {
                "attrs": {
                  "type": "object"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "type": "string"
                },
                "tag": {
                  "type": "string"
                },
                "text": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Find elements whose visible text matches a string (exact or `partial`).",
      "metadata": {},
      "name": "scrapling::find-by-text",
      "request_schema": {
        "properties": {
          "case_sensitive": {
            "type": "boolean"
          },
          "clean_match": {
            "description": "ignore surrounding/collapsing whitespace",
            "type": "boolean"
          },
          "first": {
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "limit": {
            "type": "integer"
          },
          "partial": {
            "description": "match elements that contain the text",
            "type": "boolean"
          },
          "text": {
            "type": "string"
          }
        },
        "required": [
          "html",
          "text"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "count": {
            "type": "integer"
          },
          "items": {
            "items": {
              "properties": {
                "attrs": {
                  "type": "object"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "type": "string"
                },
                "tag": {
                  "type": "string"
                },
                "text": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Structural auto-match: given one example element, return it plus similar elements.",
      "metadata": {},
      "name": "scrapling::find-similar",
      "request_schema": {
        "properties": {
          "anchor": {
            "description": "CSS selector to one example element",
            "type": "string"
          },
          "html": {
            "type": "string"
          },
          "match_text": {
            "type": "boolean"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "similarity_threshold": {
            "type": "number"
          }
        },
        "required": [
          "html",
          "anchor"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "count": {
            "type": "integer"
          },
          "items": {
            "items": {
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Internal pre_generate hook: appends scrapling usage guidance to the agent system prompt. Bound to harness::hook::pre-generate at worker startup; not called directly.",
      "metadata": {
        "internal": true
      },
      "name": "scrapling::inject-guidance",
      "request_schema": {
        "properties": {
          "generate": {
            "properties": {
              "system_prompt": {
                "description": "system prompt assembled so far",
                "type": "string"
              }
            },
            "type": "object"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "mutations": {
            "properties": {
              "system_prompt": {
                "description": "full replacement prompt (base + guidance)",
                "type": "string"
              }
            },
            "type": "object"
          }
        },
        "required": [
          "mutations"
        ],
        "type": "object"
      }
    },
    {
      "description": "Run a regex over the visible text of provided HTML; `first` returns the first match, else all.",
      "metadata": {},
      "name": "scrapling::regex",
      "request_schema": {
        "properties": {
          "first": {
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "pattern": {
            "type": "string"
          }
        },
        "required": [
          "html",
          "pattern"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "result": {
            "items": {
              "type": [
                "string",
                "null"
              ]
            },
            "type": [
              "array",
              "string",
              "null"
            ]
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Capture a page screenshot as image content blocks via a browser fetcher (dynamic or stealthy).",
      "metadata": {},
      "name": "scrapling::screenshot",
      "request_schema": {
        "properties": {
          "fetcher": {
            "enum": [
              "dynamic",
              "stealthy"
            ],
            "type": "string"
          },
          "format": {
            "enum": [
              "png",
              "jpeg"
            ],
            "type": "string"
          },
          "full_page": {
            "type": "boolean"
          },
          "headless": {
            "type": "boolean"
          },
          "network_idle": {
            "type": "boolean"
          },
          "proxy": {
            "type": "string"
          },
          "timeout": {
            "type": "number"
          },
          "url": {
            "type": "string"
          },
          "wait_selector": {
            "type": "string"
          }
        },
        "required": [
          "url"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "content": {
            "description": "image blocks (one per tile, width<=1024/height<=1536) + a text caption",
            "items": {
              "properties": {
                "data": {
                  "description": "base64 image bytes (image blocks)",
                  "type": "string"
                },
                "mime": {
                  "type": "string"
                },
                "text": {
                  "type": "string"
                },
                "type": {
                  "enum": [
                    "image",
                    "text"
                  ],
                  "type": "string"
                }
              },
              "required": [
                "type"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "mime": {
            "type": "string"
          },
          "url": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Close a session and free its browser/connection.",
      "metadata": {},
      "name": "scrapling::session-close",
      "request_schema": {
        "properties": {
          "session_id": {
            "type": "string"
          }
        },
        "required": [
          "session_id"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "closed": {
            "type": "boolean"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Fetch a URL on an open session (reuses its cookies/browser); same page/extraction output.",
      "metadata": {},
      "name": "scrapling::session-fetch",
      "request_schema": {
        "properties": {
          "css_selector": {
            "description": "scope the render to this CSS subtree (e.g. a page's content div)",
            "type": "string"
          },
          "data": {
            "type": "object"
          },
          "format": {
            "description": "render page body to this format",
            "enum": [
              "markdown",
              "text"
            ],
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "include_html": {
            "type": "boolean"
          },
          "json": {
            "type": "object"
          },
          "main_content_only": {
            "description": "strip nav/scripts/hidden before rendering",
            "type": "boolean"
          },
          "method": {
            "enum": [
              "get",
              "post",
              "put",
              "delete"
            ],
            "type": "string"
          },
          "params": {
            "type": "object"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "session_id": {
            "type": "string"
          },
          "url": {
            "type": "string"
          },
          "wait_selector": {
            "type": "string"
          }
        },
        "required": [
          "session_id",
          "url"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "captured_xhr": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "content": {
            "description": "markdown/text render when `format` requested",
            "type": "string"
          },
          "cookies": {
            "type": "object"
          },
          "encoding": {
            "type": [
              "string",
              "null"
            ]
          },
          "error": {
            "type": "string"
          },
          "extracted": {
            "type": "object"
          },
          "format": {
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "html": {
            "type": "string"
          },
          "results": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "status": {
            "type": [
              "integer",
              "null"
            ]
          },
          "url": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "List open sessions with their type and idle time.",
      "metadata": {},
      "name": "scrapling::session-list",
      "request_schema": {
        "properties": {
          "type": {
            "description": "filter by type",
            "enum": [
              "http",
              "dynamic",
              "stealthy"
            ],
            "type": "string"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "sessions": {
            "items": {
              "properties": {
                "created_at": {
                  "type": "number"
                },
                "idle_s": {
                  "type": "number"
                },
                "last_used": {
                  "type": "number"
                },
                "session_id": {
                  "type": "string"
                },
                "type": {
                  "type": "string"
                }
              },
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Open a persistent HTTP/browser session; returns a session_id that reuses cookies + state.",
      "metadata": {},
      "name": "scrapling::session-open",
      "request_schema": {
        "properties": {
          "capture_xhr": {
            "description": "regex; capture matching XHRs (browser sessions)",
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "headless": {
            "type": "boolean"
          },
          "impersonate": {
            "type": "string"
          },
          "proxies": {
            "type": "object"
          },
          "proxy": {
            "type": "string"
          },
          "real_chrome": {
            "type": "boolean"
          },
          "solve_cloudflare": {
            "type": "boolean"
          },
          "timeout": {
            "type": "number"
          },
          "type": {
            "description": "session engine",
            "enum": [
              "http",
              "dynamic",
              "stealthy"
            ],
            "type": "string"
          },
          "useragent": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "session_id": {
            "type": "string"
          },
          "type": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Camoufox stealth browser: solves Cloudflare, hardens WebRTC/canvas; extraction + bulk.",
      "metadata": {},
      "name": "scrapling::stealthy-fetch",
      "request_schema": {
        "properties": {
          "allow_webgl": {
            "type": "boolean"
          },
          "block_ads": {
            "type": "boolean"
          },
          "block_webrtc": {
            "type": "boolean"
          },
          "blocked_domains": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "capture_xhr": {
            "type": "string"
          },
          "cookies": {
            "type": "object"
          },
          "css_selector": {
            "description": "scope the render to this CSS subtree (e.g. a page's content div)",
            "type": "string"
          },
          "disable_resources": {
            "type": "boolean"
          },
          "dns_over_https": {
            "type": "boolean"
          },
          "extra_flags": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "extra_headers": {
            "type": "object"
          },
          "format": {
            "description": "render page body to this format",
            "enum": [
              "markdown",
              "text"
            ],
            "type": "string"
          },
          "google_search": {
            "type": "boolean"
          },
          "headless": {
            "type": "boolean"
          },
          "hide_canvas": {
            "type": "boolean"
          },
          "include_html": {
            "type": "boolean"
          },
          "load_dom": {
            "type": "boolean"
          },
          "locale": {
            "type": "string"
          },
          "main_content_only": {
            "description": "strip nav/scripts/hidden before rendering",
            "type": "boolean"
          },
          "max_pages": {
            "type": "integer"
          },
          "network_idle": {
            "type": "boolean"
          },
          "proxy": {
            "type": "string"
          },
          "retries": {
            "type": "integer"
          },
          "retry_delay": {
            "type": "number"
          },
          "selectors": {
            "items": {
              "properties": {
                "all": {
                  "description": "return every match as a list",
                  "type": "boolean"
                },
                "attr": {
                  "description": "extract this attribute instead of text",
                  "type": "string"
                },
                "css": {
                  "type": "string"
                },
                "html": {
                  "description": "extract inner HTML instead of text",
                  "type": "boolean"
                },
                "name": {
                  "type": "string"
                },
                "regex": {
                  "type": "string"
                },
                "xpath": {
                  "type": "string"
                }
              },
              "required": [
                "name"
              ],
              "type": "object"
            },
            "type": "array"
          },
          "solve_cloudflare": {
            "type": "boolean"
          },
          "timeout": {
            "description": "milliseconds (browser fetcher)",
            "type": "number"
          },
          "timezone_id": {
            "type": "string"
          },
          "url": {
            "type": "string"
          },
          "urls": {
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "useragent": {
            "type": "string"
          },
          "wait": {
            "description": "extra ms to wait after load",
            "type": "number"
          },
          "wait_selector": {
            "type": "string"
          },
          "wait_selector_state": {
            "enum": [
              "attached",
              "detached",
              "visible",
              "hidden"
            ],
            "type": "string"
          }
        },
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "captured_xhr": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "content": {
            "description": "markdown/text render when `format` requested",
            "type": "string"
          },
          "cookies": {
            "type": "object"
          },
          "encoding": {
            "type": [
              "string",
              "null"
            ]
          },
          "error": {
            "type": "string"
          },
          "extracted": {
            "type": "object"
          },
          "format": {
            "type": "string"
          },
          "headers": {
            "type": "object"
          },
          "html": {
            "type": "string"
          },
          "results": {
            "items": {
              "type": "object"
            },
            "type": "array"
          },
          "status": {
            "type": [
              "integer",
              "null"
            ]
          },
          "url": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "Convert HTML to compact Markdown (or text/html); optional CSS scope + main-content clean.",
      "metadata": {},
      "name": "scrapling::to-markdown",
      "request_schema": {
        "properties": {
          "css_selector": {
            "description": "convert only the subtree matching this CSS selector",
            "type": "string"
          },
          "format": {
            "enum": [
              "markdown",
              "text",
              "html"
            ],
            "type": "string"
          },
          "html": {
            "type": "string"
          },
          "main_content_only": {
            "description": "strip nav/scripts/hidden nodes first",
            "type": "boolean"
          }
        },
        "required": [
          "html"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "content": {
            "type": "string"
          },
          "format": {
            "type": "string"
          }
        },
        "type": "object"
      }
    },
    {
      "description": "One XPath query over HTML; first-or-all; `attr` pulls an attribute else text.",
      "metadata": {},
      "name": "scrapling::xpath",
      "request_schema": {
        "properties": {
          "adaptive": {
            "description": "relocate elements after a site change via saved identities",
            "type": "boolean"
          },
          "adaptive_domain": {
            "description": "page URL/domain that keys saved identities",
            "type": "string"
          },
          "attr": {
            "type": "string"
          },
          "auto_save": {
            "description": "save matched identities (defaults on when adaptive)",
            "type": "boolean"
          },
          "first": {
            "type": "boolean"
          },
          "html": {
            "type": "string"
          },
          "identifier": {
            "description": "stable key for the saved element",
            "type": "string"
          },
          "query": {
            "type": "string"
          }
        },
        "required": [
          "html",
          "query"
        ],
        "type": "object"
      },
      "response_schema": {
        "properties": {
          "result": {
            "items": {
              "type": [
                "string",
                "null"
              ]
            },
            "type": [
              "array",
              "string",
              "null"
            ]
          }
        },
        "type": "object"
      }
    }
  ],
  "triggers": []
}
```