skip to content
iii / worker
$worker

scrapling

v0.2.3

Scrapling as an iii worker — scrapling::* functions run HTTP / anti-bot / browser fetches, CSS/XPath/regex/adaptive extraction, persistent sessions, and a declarative crawl over the iii bus.

iiiverified
119 installs116 in 7d35 today
install
$iii worker add scrapling@0.2.3
bundlescrapingbrowsercrawleranti-botextractionhttp
21functions0triggers1skills0depsv0.2.3version
repoiii-hq/workers

exact versions are immutable; binary and bundle artifacts are digest-pinned.

functions

21

scrapling::crawl

function

BFS-crawl from start_urls (follow same-domain links), extract per page, stream items.

request
  • allowed_domainsstring[]

    only follow links on these hosts

  • concurrencyinteger
  • css_selectorstring

    scope the render to this CSS subtree (e.g. a page's content div)

  • download_delaynumber

    seconds to wait between crawl rounds

  • fetcherstringenum: http, stealthy, dynamic
  • formatstringenum: markdown, text

    render page body to this format

  • impersonatestring
  • include_htmlboolean
  • main_content_onlyboolean

    strip nav/scripts/hidden before rendering

  • max_depthinteger
  • max_pagesinteger
  • same_domainboolean

    follow only same-host links (default true)

  • selectorsobject[]
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
  • start_urlsstring[]
  • stream_namestring

    stream to emit items on (default scrapling::crawl)

  • urlstring

    single start URL (alternative to start_urls)

response
  • itemsobject[]

    a small sample of streamed items

    empty object
  • statsobject
    • crawledinteger
    • errorsinteger
    • itemsinteger
    • stoppedstring
  • streamobject

    read the full item stream via stream::on with this name + group_id

    • group_idstring
    • namestring

scrapling::css

function

One CSS query over HTML; first-or-all; `attr` pulls an attribute else text.

request
  • adaptiveboolean

    relocate elements after a site change via saved identities

  • adaptive_domainstring

    page URL/domain that keys saved identities

  • attrstring
  • auto_saveboolean

    save matched identities (defaults on when adaptive)

  • firstboolean
  • htmlstringrequired
  • identifierstring

    stable key for the saved element

  • querystringrequired
response
  • resultstring[]

scrapling::describe

function

Describe the first css/xpath match: attrs, generated selectors, class list, DOM context.

request
  • htmlstringrequired
  • kindstringenum: css, xpath
  • querystringrequired
response
  • elementobject
    • attrsobject
    • childreninteger
    • classesstring[]
    • cssstring
    • full_cssstring
    • full_xpathstring
    • htmlstring
    • parent_tagstring
    • siblingsinteger
    • tagstring
    • textstring
    • xpathstring
  • foundboolean

scrapling::dynamic-fetch

function

Playwright/Chromium fetch: JS render, waits, XHR capture, CDP; extraction + bulk.

request
  • block_adsboolean
  • blocked_domainsstring[]
  • capture_xhrstring
  • cdp_urlstring
  • cookiesobject
  • css_selectorstring

    scope the render to this CSS subtree (e.g. a page's content div)

  • disable_resourcesboolean
  • dns_over_httpsboolean
  • extra_flagsstring[]
  • extra_headersobject
  • formatstringenum: markdown, text

    render page body to this format

  • google_searchboolean
  • headlessboolean
  • include_htmlboolean
  • load_domboolean
  • localestring
  • main_content_onlyboolean

    strip nav/scripts/hidden before rendering

  • max_pagesinteger
  • network_idleboolean
  • proxystring
  • real_chromeboolean
  • retriesinteger
  • retry_delaynumber
  • selectorsobject[]
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
  • timeoutnumber

    milliseconds (browser fetcher)

  • timezone_idstring
  • urlstring
  • urlsstring[]
  • useragentstring
  • waitnumber

    extra ms to wait after load

  • wait_selectorstring
  • wait_selector_statestringenum: attached, detached, visible, hidden
response
  • captured_xhrobject[]
    empty object
  • contentstring

    markdown/text render when `format` requested

  • cookiesobject
  • encodingstring
  • errorstring
  • extractedobject
  • formatstring
  • headersobject
  • htmlstring
  • resultsobject[]
    empty object
  • statusinteger
  • urlstring

scrapling::extract

function

Parse HTML with a selector list (css/xpath/regex, text/attr/html, all-or-first).

request
  • adaptiveboolean

    relocate elements after a site change via saved identities

  • adaptive_domainstring

    page URL/domain that keys saved identities

  • auto_saveboolean

    save matched identities (defaults on when adaptive)

  • htmlstringrequired
  • selectorsobject[]required
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
response
  • extractedobject

scrapling::fetch

function

Fast HTTP fetch, TLS impersonation: get/post/put/delete, inline extraction, bulk `urls`.

request
  • cookiesobject
  • css_selectorstring

    scope the render to this CSS subtree (e.g. a page's content div)

  • dataobject
  • follow_redirectsboolean
  • formatstringenum: markdown, text

    render page body to this format

  • headersobject
  • http3boolean
  • impersonatestring

    TLS/UA fingerprint, e.g. 'chrome'

  • include_htmlboolean
  • jsonobject
  • main_content_onlyboolean

    strip nav/scripts/hidden before rendering

  • max_redirectsinteger
  • methodstringenum: get, post, put, delete
  • paramsobject
  • proxiesobject

    per-scheme proxies, e.g. {"https": "http://..."}

  • proxystring
  • proxy_authstring[]

    [user, password]

  • retriesinteger
  • retry_delaynumber
  • selectorsobject[]
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
  • stealthy_headersboolean
  • timeoutnumber

    seconds (HTTP fetcher)

  • urlstring
  • urlsstring[]
  • verifyboolean
response
  • captured_xhrobject[]
    empty object
  • contentstring

    markdown/text render when `format` requested

  • cookiesobject
  • encodingstring
  • errorstring
  • extractedobject
  • formatstring
  • headersobject
  • htmlstring
  • resultsobject[]
    empty object
  • statusinteger
  • urlstring

scrapling::find

function

Find elements by tag/attribute filters (+ optional text regex); BeautifulSoup-style.

request
  • attrsobject

    attribute filters, e.g. {"class": "card"}

  • firstboolean
  • htmlstringrequired
  • limitinteger
  • tagstring

    tag name or list of tag names

  • text_regexstring

    keep only elements whose text matches this regex

response
  • countinteger
  • itemsobject[]
    • attrsobject
    • cssstring
    • htmlstring
    • tagstring
    • textstring
    • xpathstring

scrapling::find-by-regex

function

Find elements whose visible text matches a regex pattern.

request
  • case_sensitiveboolean
  • clean_matchboolean
  • firstboolean
  • htmlstringrequired
  • limitinteger
  • patternstringrequired
response
  • countinteger
  • itemsobject[]
    • attrsobject
    • cssstring
    • htmlstring
    • tagstring
    • textstring
    • xpathstring

scrapling::find-by-text

function

Find elements whose visible text matches a string (exact or `partial`).

request
  • case_sensitiveboolean
  • clean_matchboolean

    ignore surrounding/collapsing whitespace

  • firstboolean
  • htmlstringrequired
  • limitinteger
  • partialboolean

    match elements that contain the text

  • textstringrequired
response
  • countinteger
  • itemsobject[]
    • attrsobject
    • cssstring
    • htmlstring
    • tagstring
    • textstring
    • xpathstring

scrapling::find-similar

function

Structural auto-match: given one example element, return it plus similar elements.

request
  • anchorstringrequired

    CSS selector to one example element

  • htmlstringrequired
  • match_textboolean
  • selectorsobject[]
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
  • similarity_thresholdnumber
response
  • countinteger
  • itemsobject[]
    empty object

scrapling::inject-guidance

function

Internal pre_generate hook: appends scrapling usage guidance to the agent system prompt. Bound to harness::hook::pre-generate at worker startup; not called directly.

request
  • generateobject
    • system_promptstring

      system prompt assembled so far

response
  • mutationsobjectrequired
    • system_promptstring

      full replacement prompt (base + guidance)

scrapling::on-registry-changed

function

Internal: on engine registry changes, bind the scrapling inject-guidance pre-generate hook once the harness has registered its trigger type. Not called directly.

request
valuenull
response
  • okbooleanrequired

scrapling::regex

function

Run a regex over the visible text of provided HTML; `first` returns the first match, else all.

request
  • firstboolean
  • htmlstringrequired
  • patternstringrequired
response
  • resultstring[]

scrapling::screenshot

function

Capture a page screenshot as image content blocks via a browser fetcher (dynamic or stealthy).

request
  • fetcherstringenum: dynamic, stealthy
  • formatstringenum: png, jpeg
  • full_pageboolean
  • headlessboolean
  • network_idleboolean
  • proxystring
  • timeoutnumber
  • urlstringrequired
  • wait_selectorstring
response
  • contentobject[]

    image blocks (one per tile, width<=1024/height<=1536) + a text caption

    • datastring

      base64 image bytes (image blocks)

    • mimestring
    • textstring
    • typestringrequiredenum: image, text
  • mimestring
  • urlstring

scrapling::session-close

function

Close a session and free its browser/connection.

request
  • session_idstringrequired
response
  • closedboolean

scrapling::session-fetch

function

Fetch a URL on an open session (reuses its cookies/browser); same page/extraction output.

request
  • css_selectorstring

    scope the render to this CSS subtree (e.g. a page's content div)

  • dataobject
  • formatstringenum: markdown, text

    render page body to this format

  • headersobject
  • include_htmlboolean
  • jsonobject
  • main_content_onlyboolean

    strip nav/scripts/hidden before rendering

  • methodstringenum: get, post, put, delete
  • paramsobject
  • selectorsobject[]
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
  • session_idstringrequired
  • urlstringrequired
  • wait_selectorstring
response
  • captured_xhrobject[]
    empty object
  • contentstring

    markdown/text render when `format` requested

  • cookiesobject
  • encodingstring
  • errorstring
  • extractedobject
  • formatstring
  • headersobject
  • htmlstring
  • resultsobject[]
    empty object
  • statusinteger
  • urlstring

scrapling::session-list

function

List open sessions with their type and idle time.

request
  • typestringenum: http, dynamic, stealthy

    filter by type

response
  • sessionsobject[]
    • created_atnumber
    • idle_snumber
    • last_usednumber
    • session_idstring
    • typestring

scrapling::session-open

function

Open a persistent HTTP/browser session; returns a session_id that reuses cookies + state.

request
  • capture_xhrstring

    regex; capture matching XHRs (browser sessions)

  • headersobject
  • headlessboolean
  • impersonatestring
  • proxiesobject
  • proxystring
  • real_chromeboolean
  • solve_cloudflareboolean
  • timeoutnumber
  • typestringenum: http, dynamic, stealthy

    session engine

  • useragentstring
response
  • session_idstring
  • typestring

scrapling::stealthy-fetch

function

Camoufox stealth browser: solves Cloudflare, hardens WebRTC/canvas; extraction + bulk.

request
  • allow_webglboolean
  • block_adsboolean
  • block_webrtcboolean
  • blocked_domainsstring[]
  • capture_xhrstring
  • cookiesobject
  • css_selectorstring

    scope the render to this CSS subtree (e.g. a page's content div)

  • disable_resourcesboolean
  • dns_over_httpsboolean
  • extra_flagsstring[]
  • extra_headersobject
  • formatstringenum: markdown, text

    render page body to this format

  • google_searchboolean
  • headlessboolean
  • hide_canvasboolean
  • include_htmlboolean
  • load_domboolean
  • localestring
  • main_content_onlyboolean

    strip nav/scripts/hidden before rendering

  • max_pagesinteger
  • network_idleboolean
  • proxystring
  • retriesinteger
  • retry_delaynumber
  • selectorsobject[]
    • allboolean

      return every match as a list

    • attrstring

      extract this attribute instead of text

    • cssstring
    • htmlboolean

      extract inner HTML instead of text

    • namestringrequired
    • regexstring
    • xpathstring
  • solve_cloudflareboolean
  • timeoutnumber

    milliseconds (browser fetcher)

  • timezone_idstring
  • urlstring
  • urlsstring[]
  • useragentstring
  • waitnumber

    extra ms to wait after load

  • wait_selectorstring
  • wait_selector_statestringenum: attached, detached, visible, hidden
response
  • captured_xhrobject[]
    empty object
  • contentstring

    markdown/text render when `format` requested

  • cookiesobject
  • encodingstring
  • errorstring
  • extractedobject
  • formatstring
  • headersobject
  • htmlstring
  • resultsobject[]
    empty object
  • statusinteger
  • urlstring

scrapling::to-markdown

function

Convert HTML to compact Markdown (or text/html); optional CSS scope + main-content clean.

request
  • css_selectorstring

    convert only the subtree matching this CSS selector

  • formatstringenum: markdown, text, html
  • htmlstringrequired
  • main_content_onlyboolean

    strip nav/scripts/hidden nodes first

response
  • contentstring
  • formatstring

scrapling::xpath

function

One XPath query over HTML; first-or-all; `attr` pulls an attribute else text.

request
  • adaptiveboolean

    relocate elements after a site change via saved identities

  • adaptive_domainstring

    page URL/domain that keys saved identities

  • attrstring
  • auto_saveboolean

    save matched identities (defaults on when adaptive)

  • firstboolean
  • htmlstringrequired
  • identifierstring

    stable key for the saved element

  • querystringrequired
response
  • resultstring[]

triggers

0
no triggers registered