SignalWire DocsFast web scraping and crawling. Extracts text, markdown, or structured data
from any public URL, optionally following links up to a bounded depth. Uses
cheerio for parsing and enforces an SSRF guard on crawl hops.
Class: SpiderSkill
Tools: scrape_url, crawl_site, extract_structured_data (each is
prefixed with <tool_name>_ when tool_name is set).
Required packages: cheerio
Env vars: SWML_ALLOW_PRIVATE_URLS=true relaxes the SSRF guard for local
testing.
Multi-instance: yes — set a distinct tool_name per instance.
Prefix prepended to each emitted tool name (e.g., tool_name="news" gives
news_scrape_url, news_crawl_site, news_extract_structured_data).
Required when registering multiple instances on the same agent.
Delay between requests in seconds (minimum 0).
Number of concurrent requests allowed (range 1-20).
Per-request timeout in seconds (range 1-60).
Maximum number of pages to scrape (range 1-100).
Maximum crawl depth. 0 restricts to a single page; range 0-5.
Content extraction method. One of "fast_text", "clean_text",
"full_text", "html", "markdown", "structured", "custom". Only
fast_text, markdown, and structured are wired through the handlers
in the TypeScript port; the others fall back to fast_text.
Maximum extracted text length in characters (range 100-100000).
Whether to clean extracted text (trim whitespace, collapse runs, etc.).
Map of name → CSS selector used for structured extraction.
URL patterns to follow when crawling.
User-Agent header for outbound requests. Defaults to a Chrome-compatible UA string.
Additional HTTP headers sent with each request.
Whether to respect robots.txt. Defaults to false to match Python’s
runtime behavior.
Whether to cache scraped pages in memory.