$worker

scrapling

Name: scrapling
Author: iii

v0.2.5

Scrapling as an iii worker — scrapling::* functions run HTTP / anti-bot / browser fetches, CSS/XPath/regex/adaptive extraction, persistent sessions, and a declarative crawl over the iii bus.

iiiverified

118 installs115 in 7d34 today

install

$iii worker add scrapling@0.2.5

bundlescraping browser crawler anti-bot extraction http

20functions 0triggers 1skills 0deps v0.2.5version

repoiii-hq/workers

exact versions are immutable; binary and bundle artifacts are digest-pinned.

functions

scrapling::crawl

function

BFS-crawl from start_urls (follow same-domain links), extract per page, stream items.

request

allowed_domainsstring[]
only follow links on these hosts
concurrencyinteger
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
download_delaynumber
seconds to wait between crawl rounds
fetcherstringenum: http, stealthy, dynamic
formatstringenum: markdown, text
render page body to this format
impersonatestring
include_htmlboolean
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_depthinteger
max_pagesinteger
same_domainboolean
follow only same-host links (default true)
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
start_urlsstring[]
stream_namestring
stream to emit items on (default scrapling::crawl)
urlstring
single start URL (alternative to start_urls)

response

itemsobject[]
a small sample of streamed items
empty object
statsobject
- crawledinteger
- errorsinteger
- itemsinteger
- stoppedstring
streamobject
read the full item stream via stream::on with this name + group_id
- group_idstring
- namestring

scrapling::css

function

One CSS query over HTML; first-or-all; `attr` pulls an attribute else text.

request

adaptiveboolean
relocate elements after a site change via saved identities
adaptive_domainstring
page URL/domain that keys saved identities
attrstring
auto_saveboolean
save matched identities (defaults on when adaptive)
firstboolean
htmlstringrequired
identifierstring
stable key for the saved element
querystringrequired

response

resultstring[]

scrapling::describe

function

Describe the first css/xpath match: attrs, generated selectors, class list, DOM context.

request

htmlstringrequired
kindstringenum: css, xpath
querystringrequired

response

elementobject
- attrsobject
- childreninteger
- classesstring[]
- cssstring
- full_cssstring
- full_xpathstring
- htmlstring
- parent_tagstring
- siblingsinteger
- tagstring
- textstring
- xpathstring
foundboolean

scrapling::dynamic-fetch

function

Playwright/Chromium fetch: JS render, waits, XHR capture, CDP; extraction + bulk.

request

block_adsboolean
blocked_domainsstring[]
capture_xhrstring
cdp_urlstring
cookiesobject
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
disable_resourcesboolean
dns_over_httpsboolean
extra_flagsstring[]
extra_headersobject
formatstringenum: markdown, text
render page body to this format
google_searchboolean
headlessboolean
include_htmlboolean
load_domboolean
localestring
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_pagesinteger
network_idleboolean
proxystring
real_chromeboolean
retriesinteger
retry_delaynumber
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
timeoutnumber
milliseconds (browser fetcher)
timezone_idstring
urlstring
urlsstring[]
useragentstring
waitnumber
extra ms to wait after load
wait_selectorstring
wait_selector_statestringenum: attached, detached, visible, hidden

response

captured_xhrobject[]
empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
empty object
statusinteger
urlstring

scrapling::extract

function

Parse HTML with a selector list (css/xpath/regex, text/attr/html, all-or-first).

request

adaptiveboolean
relocate elements after a site change via saved identities
adaptive_domainstring
page URL/domain that keys saved identities
auto_saveboolean
save matched identities (defaults on when adaptive)
htmlstringrequired
selectorsobject[]required
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring

response

extractedobject

scrapling::fetch

function

Fast HTTP fetch, TLS impersonation: get/post/put/delete, inline extraction, bulk `urls`.

request

cookiesobject
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
dataobject
follow_redirectsboolean
formatstringenum: markdown, text
render page body to this format
headersobject
http3boolean
impersonatestring
TLS/UA fingerprint, e.g. 'chrome'
include_htmlboolean
jsonobject
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_redirectsinteger
methodstringenum: get, post, put, delete
paramsobject
proxiesobject
per-scheme proxies, e.g. {"https": "http://..."}
proxystring
proxy_authstring[]
[user, password]
retriesinteger
retry_delaynumber
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
stealthy_headersboolean
timeoutnumber
seconds (HTTP fetcher)
urlstring
urlsstring[]
verifyboolean

response

captured_xhrobject[]
empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
empty object
statusinteger
urlstring

scrapling::find

function

Find elements by tag/attribute filters (+ optional text regex); BeautifulSoup-style.

request

attrsobject
attribute filters, e.g. {"class": "card"}
firstboolean
htmlstringrequired
limitinteger
tagstring
tag name or list of tag names
text_regexstring
keep only elements whose text matches this regex

response

countinteger
itemsobject[]
- attrsobject
- cssstring
- htmlstring
- tagstring
- textstring
- xpathstring

scrapling::find-by-regex

function

Find elements whose visible text matches a regex pattern.

request

case_sensitiveboolean
clean_matchboolean
firstboolean
htmlstringrequired
limitinteger
patternstringrequired

response

countinteger
itemsobject[]
- attrsobject
- cssstring
- htmlstring
- tagstring
- textstring
- xpathstring

scrapling::find-by-text

function

Find elements whose visible text matches a string (exact or `partial`).

request

case_sensitiveboolean
clean_matchboolean
ignore surrounding/collapsing whitespace
firstboolean
htmlstringrequired
limitinteger
partialboolean
match elements that contain the text
textstringrequired

response

countinteger
itemsobject[]
- attrsobject
- cssstring
- htmlstring
- tagstring
- textstring
- xpathstring

scrapling::find-similar

function

Structural auto-match: given one example element, return it plus similar elements.

request

anchorstringrequired
CSS selector to one example element
htmlstringrequired
match_textboolean
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
similarity_thresholdnumber

response

countinteger
itemsobject[]
empty object

scrapling::inject-guidance

function

Internal pre_generate hook: appends scrapling usage guidance to the agent system prompt. Bound to harness::hook::pre-generate at worker startup; not called directly.

request

generateobject
- system_promptstring
  system prompt assembled so far

response

mutationsobjectrequired
- system_promptstring
  full replacement prompt (base + guidance)

scrapling::regex

function

Run a regex over the visible text of provided HTML; `first` returns the first match, else all.

request

firstboolean
htmlstringrequired
patternstringrequired

response

resultstring[]

scrapling::screenshot

function

Capture a page screenshot as image content blocks via a browser fetcher (dynamic or stealthy).

request

fetcherstringenum: dynamic, stealthy
formatstringenum: png, jpeg
full_pageboolean
headlessboolean
network_idleboolean
proxystring
timeoutnumber
urlstringrequired
wait_selectorstring

response

contentobject[]
image blocks (one per tile, width<=1024/height<=1536) + a text caption
- datastring
  base64 image bytes (image blocks)
- mimestring
- textstring
- typestringrequiredenum: image, text
mimestring
urlstring

scrapling::session-close

function

Close a session and free its browser/connection.

request

session_idstringrequired

response

closedboolean

scrapling::session-fetch

function

Fetch a URL on an open session (reuses its cookies/browser); same page/extraction output.

request

css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
dataobject
formatstringenum: markdown, text
render page body to this format
headersobject
include_htmlboolean
jsonobject
main_content_onlyboolean
strip nav/scripts/hidden before rendering
methodstringenum: get, post, put, delete
paramsobject
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
session_idstringrequired
urlstringrequired
wait_selectorstring

response

captured_xhrobject[]
empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
empty object
statusinteger
urlstring

scrapling::session-list

function

List open sessions with their type and idle time.

request

typestringenum: http, dynamic, stealthy
filter by type

response

sessionsobject[]
- created_atnumber
- idle_snumber
- last_usednumber
- session_idstring
- typestring

scrapling::session-open

function

Open a persistent HTTP/browser session; returns a session_id that reuses cookies + state.

request

capture_xhrstring
regex; capture matching XHRs (browser sessions)
headersobject
headlessboolean
impersonatestring
proxiesobject
proxystring
real_chromeboolean
solve_cloudflareboolean
timeoutnumber
typestringenum: http, dynamic, stealthy
session engine
useragentstring

response

session_idstring
typestring

scrapling::stealthy-fetch

function

Camoufox stealth browser: solves Cloudflare, hardens WebRTC/canvas; extraction + bulk.

request

allow_webglboolean
block_adsboolean
block_webrtcboolean
blocked_domainsstring[]
capture_xhrstring
cookiesobject
css_selectorstring
scope the render to this CSS subtree (e.g. a page's content div)
disable_resourcesboolean
dns_over_httpsboolean
extra_flagsstring[]
extra_headersobject
formatstringenum: markdown, text
render page body to this format
google_searchboolean
headlessboolean
hide_canvasboolean
include_htmlboolean
load_domboolean
localestring
main_content_onlyboolean
strip nav/scripts/hidden before rendering
max_pagesinteger
network_idleboolean
proxystring
retriesinteger
retry_delaynumber
selectorsobject[]
- allboolean
  return every match as a list
- attrstring
  extract this attribute instead of text
- cssstring
- htmlboolean
  extract inner HTML instead of text
- namestringrequired
- regexstring
- xpathstring
solve_cloudflareboolean
timeoutnumber
milliseconds (browser fetcher)
timezone_idstring
urlstring
urlsstring[]
useragentstring
waitnumber
extra ms to wait after load
wait_selectorstring
wait_selector_statestringenum: attached, detached, visible, hidden

response

captured_xhrobject[]
empty object
contentstring
markdown/text render when `format` requested
cookiesobject
encodingstring
errorstring
extractedobject
formatstring
headersobject
htmlstring
resultsobject[]
empty object
statusinteger
urlstring

scrapling::to-markdown

function

Convert HTML to compact Markdown (or text/html); optional CSS scope + main-content clean.

request

css_selectorstring
convert only the subtree matching this CSS selector
formatstringenum: markdown, text, html
htmlstringrequired
main_content_onlyboolean
strip nav/scripts/hidden nodes first

response

contentstring
formatstring

scrapling::xpath

function

One XPath query over HTML; first-or-all; `attr` pulls an attribute else text.

request

adaptiveboolean
relocate elements after a site change via saved identities
adaptive_domainstring
page URL/domain that keys saved identities
attrstring
auto_saveboolean
save matched identities (defaults on when adaptive)
firstboolean
htmlstringrequired
identifierstring
stable key for the saved element
querystringrequired

response

resultstring[]

triggers

no triggers registered