SpiderSkill | SignalWire

Fast web scraping and crawling. Extracts text, markdown, or structured data from any public URL, optionally following links up to a bounded depth. Uses cheerio for parsing and enforces an SSRF guard: the URL you pass and every redirect it follows must resolve to a public address.

Class: SpiderSkill

Tools: scrape_url, crawl_site, extract_structured_data (each is prefixed with <tool_name>_ when tool_name is set).

Required packages: cheerio

Env vars: SWML_ALLOW_PRIVATE_URLS=true relaxes the SSRF guard for local testing.

Multi-instance: yes — set a distinct tool_name per instance.

tool_name

string

Prefix prepended to each emitted tool name (e.g., tool_name="news" gives news_scrape_url, news_crawl_site, news_extract_structured_data). Required when registering multiple instances on the same agent.

delay

numberDefaults to 0.1

Delay between requests in seconds (minimum 0).

concurrent_requests

integerDefaults to 5

Deprecated, and has no effect: the spider fetches one page at a time. Setting it logs a warning (range 1-20).

timeout

integerDefaults to 5

Per-request timeout in seconds (range 1-60).

max_pages

integerDefaults to 1

Maximum number of pages to scrape (range 1-100).

max_depth

integerDefaults to 0

Maximum crawl depth. 0 restricts to a single page; range 0-5.

extract_type

stringDefaults to fast_text

Content extraction method. One of "fast_text", "markdown", or "structured". The values "clean_text", "full_text", "html", and "custom" were never implemented: they still work as fast_text and log a warning. Any other value makes setup() fail.

max_text_length

integerDefaults to 3000

Maximum extracted text length in characters (range 100-100000).

clean_text

booleanDefaults to true

Whether to clean extracted text (trim whitespace, collapse runs, etc.).

selectors

Record<string, string>Defaults to {}

Map of name → CSS selector used for structured extraction.

follow_patterns

string[]Defaults to []

URL patterns to follow when crawling.

user_agent

stringDefaults to Spider/1.0 (SignalWire AI Agent)

User-Agent header for outbound requests. Also the agent name matched against robots.txt rules when follow_robots_txt is true.

headers

Record<string, string>Defaults to {}

Additional HTTP headers sent with each request.

follow_robots_txt

booleanDefaults to false

Skip pages that the site’s robots.txt disallows for user_agent. This covers each page the tools fetch and each redirect they follow. A robots.txt answered with 401 or 403 disallows every page, and any other 4xx allows every page. A server error or a failed request disallows the page and is retried on the next request. Rules are kept for 24 hours.

cache_enabled

booleanDefaults to true

Whether to cache scraped pages in memory.

Example

import { AgentBase, SpiderSkill } from '@signalwire/sdk';
const agent = new AgentBase({ name: 'assistant', route: '/assistant' });
agent.setPromptText('You are a research assistant.');
await agent.addSkill(new SpiderSkill({
  extract_type: 'markdown',
  max_pages: 5,
  max_depth: 1,
  follow_patterns: ['/docs/'],
}));
agent.run();