Crw Crawl
Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.
Install
/plugin marketplace add fastcrw/crwThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add fastcrw/crw --skill crw-crawlgit clone --depth 1 https://github.com/fastcrw/crw.git /tmp/crw
mkdir -p ~/.claude/skills
cp -r /tmp/crw/skills/crw-crawl ~/.claude/skills/crw-crawlPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# crw-crawl — bulk page extraction
## When to use
- You need content from **many pages** under a site or section, not just one.
- Step 4 in the [crw ladder](../crw/SKILL.md): if you only need a handful of
known URLs, use [crw-scrape](../crw-scrape/SKILL.md) in a loop instead —
it's simpler and gives you content immediately. Use crawl when the set of
URLs is unknown or large.
- Always **map first** ([crw-map](../crw-map/SKILL.md)) to estimate page
count before committing. A misconfigured crawl on a 50 000-page site is
expensive; a map call is cheap.
- Start conservative: `depth 1, limit 10`. Scale up once you verify scope.
## Quick start
**CLI** (synchronous streaming output):
```bash
crw crawl "https://docs.example.com" -d 2 -l 50 # markdown to stdout
crw crawl "https://docs.example.com/api" -d 1 -l 20 --format json
crw crawl "https://example.com" --js --rate-limit 1.0 --concurrency 3
```
**MCP** (async — returns a job ID, poll for results):
```
# Start the crawl
crw_crawl(url="https://docs.example.com", maxDepth=2, maxPages=50)
→ { "id": "a1b2c3d4-..." }
# Poll until status == "completed"
crw_check_crawl_status(id="a1b2c3d4-...")
→ { "status": "scraping|completed|failed", "data": [...] }
```
**REST** (async — POST to start, GET to poll, DELETE to cancel):Details
- Skill name
crw-crawl- Repository
- fastcrw/crw
- Path
skills/crw-crawl/SKILL.md- License
- AGPL-3.0
- Repo updated
- 18 November 2026
- Listed in
- heilcheng/awesome-agent-skills
FAQ
What does the Crw Crawl skill do?
Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.
How do I install the Crw Crawl skill?
Claude Code plugin: /plugin marketplace add fastcrw/crw. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Crw Crawl popular?
It lives in fastcrw/crw, which has 1,268 GitHub stars and 102 forks, and is included in 1 curated skill list.
Related skills
Crw
1.3kfastcrw/crw
Scrape, crawl, map, and search the web using fastCRW's native /v1 API. Use when the user needs web page content, site-wide extraction, URL discovery, or web search results. Single binary, 14 MB RAM; /v2 exists separately for Firecrawl migration.
Crw
1.3kfastcrw/crw
Scrape, crawl, map, search, parse, and extract web data with fastCRW — the open-source, self-hostable Firecrawl alternative (single Rust binary, ~14 MB RAM, Firecrawl-compatible /v1 + /v2 API). Use whenever the user needs page content, site-wide extraction, URL discovery, web search, PDF parsing, structured JSON from pages, or change tracking. Also use when the user mentions Firecrawl, Tavily, Crawl4AI, or "scrape/crawl/map/fetch/get the page/read this site/search the web" — crw is a drop-in for the Firecrawl SDKs.
Crw Best Practices
1.3kfastcrw/crw
Reference skill for building production-ready crw integrations. Covers verb selection, call surfaces (CLI/MCP/REST), post-filtering strategies, context-window hygiene, Hybrid RAG patterns, common pitfalls, and crw-specific operational considerations (search backend limits, renderer pool, proxy rotation). Load this when writing application code that embeds crw, designing a multi-step agent workflow, or debugging an integration that isn't behaving as expected.
Crw Dynamic Search
1.3kfastcrw/crw
Programmatic web search and scrape with context isolation. Use for any research task where you need to search the web, filter results, and extract specific information — without flooding your context window with raw HTML and boilerplate. This is the single biggest token-saver in the crw skill set. Triggered by "search for", "look up", "find", "research", "what's the latest on", or any query that requires current web information. Also use when asked to "search and filter", "find the important parts", or any task where you suspect the raw output will be large (multi-page scrapes, news aggregatio
Crw Map
1.3kfastcrw/crw
Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.
Qual Quant Triangulation
3.4kOwl-Listener/designer-skills
Reconcile what the numbers say with what users say, and design the study that settles it rather than restates it. Use when behavioural data and research findings point different ways. For reading the data on its own, use `behavioural-analytics`; for synthesising interviews on their own, use `affinity-diagram`.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.