Crw Crawl

Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.

1,268 stars 102 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add fastcrw/crw

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add fastcrw/crw --skill crw-crawl
Manual install
git clone --depth 1 https://github.com/fastcrw/crw.git /tmp/crw
mkdir -p ~/.claude/skills
cp -r /tmp/crw/skills/crw-crawl ~/.claude/skills/crw-crawl

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# crw-crawl — bulk page extraction

## When to use

- You need content from **many pages** under a site or section, not just one.
- Step 4 in the [crw ladder](../crw/SKILL.md): if you only need a handful of
  known URLs, use [crw-scrape](../crw-scrape/SKILL.md) in a loop instead —
  it's simpler and gives you content immediately. Use crawl when the set of
  URLs is unknown or large.
- Always **map first** ([crw-map](../crw-map/SKILL.md)) to estimate page
  count before committing. A misconfigured crawl on a 50 000-page site is
  expensive; a map call is cheap.
- Start conservative: `depth 1, limit 10`. Scale up once you verify scope.

## Quick start

**CLI** (synchronous streaming output):
```bash
crw crawl "https://docs.example.com" -d 2 -l 50         # markdown to stdout
crw crawl "https://docs.example.com/api" -d 1 -l 20 --format json
crw crawl "https://example.com" --js --rate-limit 1.0 --concurrency 3
```

**MCP** (async — returns a job ID, poll for results):
```
# Start the crawl
crw_crawl(url="https://docs.example.com", maxDepth=2, maxPages=50)
→ { "id": "a1b2c3d4-..." }

# Poll until status == "completed"
crw_check_crawl_status(id="a1b2c3d4-...")
→ { "status": "scraping|completed|failed", "data": [...] }
```

**REST** (async — POST to start, GET to poll, DELETE to cancel):
Read the full skill on GitHub

Details

Skill name
crw-crawl
Repository
fastcrw/crw
Path
skills/crw-crawl/SKILL.md
License
AGPL-3.0
Repo updated
18 November 2026

FAQ

What does the Crw Crawl skill do?

Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.

How do I install the Crw Crawl skill?

Claude Code plugin: /plugin marketplace add fastcrw/crw. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Crw Crawl popular?

It lives in fastcrw/crw, which has 1,268 GitHub stars and 102 forks, and is included in 1 curated skill list.

Related skills

Crw

1.3k

fastcrw/crw

Scrape, crawl, map, and search the web using fastCRW's native /v1 API. Use when the user needs web page content, site-wide extraction, URL discovery, or web search results. Single binary, 14 MB RAM; /v2 exists separately for Firecrawl migration.

Writing & ContentIn 1 list102

Crw

1.3k

fastcrw/crw

Scrape, crawl, map, search, parse, and extract web data with fastCRW — the open-source, self-hostable Firecrawl alternative (single Rust binary, ~14 MB RAM, Firecrawl-compatible /v1 + /v2 API). Use whenever the user needs page content, site-wide extraction, URL discovery, web search, PDF parsing, structured JSON from pages, or change tracking. Also use when the user mentions Firecrawl, Tavily, Crawl4AI, or "scrape/crawl/map/fetch/get the page/read this site/search the web" — crw is a drop-in for the Firecrawl SDKs.

Data & AnalyticsIn 1 list102

Crw Best Practices

1.3k

fastcrw/crw

Reference skill for building production-ready crw integrations. Covers verb selection, call surfaces (CLI/MCP/REST), post-filtering strategies, context-window hygiene, Hybrid RAG patterns, common pitfalls, and crw-specific operational considerations (search backend limits, renderer pool, proxy rotation). Load this when writing application code that embeds crw, designing a multi-step agent workflow, or debugging an integration that isn't behaving as expected.

AI, LLMs & AgentsIn 1 list102

Crw Dynamic Search

1.3k

fastcrw/crw

Programmatic web search and scrape with context isolation. Use for any research task where you need to search the web, filter results, and extract specific information — without flooding your context window with raw HTML and boilerplate. This is the single biggest token-saver in the crw skill set. Triggered by "search for", "look up", "find", "research", "what's the latest on", or any query that requires current web information. Also use when asked to "search and filter", "find the important parts", or any task where you suspect the raw output will be large (multi-page scrapes, news aggregatio

Data & AnalyticsIn 1 list102

Crw Map

1.3k

fastcrw/crw

Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.

Data & AnalyticsIn 1 list102

Qual Quant Triangulation

3.4k

Owl-Listener/designer-skills

Reconcile what the numbers say with what users say, and design the study that settles it rather than restates it. Use when behavioural data and research findings point different ways. For reading the data on its own, use `behavioural-analytics`; for synthesising interviews on their own, use `affinity-diagram`.

Data & Analytics447

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.