Crw Map

Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.

1,268 stars 102 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add fastcrw/crw

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add fastcrw/crw --skill crw-map
Manual install
git clone --depth 1 https://github.com/fastcrw/crw.git /tmp/crw
mkdir -p ~/.claude/skills
cp -r /tmp/crw/skills/crw-map ~/.claude/skills/crw-map

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# crw-map — URL discovery without content

## When to use

- You need to know **which URLs exist** on a site before committing to a
  crawl — map first, then crawl only the section you need.
- Step 3 in the [crw ladder](../crw/SKILL.md): if you want content, combine
  with [crw-scrape](../crw-scrape/SKILL.md) (single pages) or
  [crw-crawl](../crw-crawl/SKILL.md) (bulk). Map is URL-only — no page
  content is returned.
- Estimating crawl cost: `crw map docs.example.com | wc -l` tells you how
  many pages a subsequent crawl would touch before you commit.
- Finding a specific path: filter the URL list with `grep` instead of
  crawling the whole site.

## Quick start

**CLI** (binary on PATH):
```bash
crw map "https://docs.example.com"                       # URLs to stdout
crw map "https://docs.example.com" -d 3 --format json   # JSON object, depth 3
crw map "https://example.com" --sitemap-only             # sitemap.xml only
crw map "https://example.com" --no-sitemap               # link crawl only
crw map "https://example.com" --format json > .crw/urls.json
```

**MCP** (inside an agent harness):
```
crw_map(url="https://docs.example.com")
crw_map(url="https://docs.example.com", maxDepth=3, limit=200)
crw_map(url="https://example.com", useSitemap=false)       # link crawl only
crw_map(url="https://example.com", crawlFallback=false)    # sitemap only
```

**REST** (drop-in for Firecrawl SDKs — just swap the base URL):
Read the full skill on GitHub

Details

Skill name
crw-map
Repository
fastcrw/crw
Path
skills/crw-map/SKILL.md
License
AGPL-3.0
Repo updated
18 November 2026

FAQ

What does the Crw Map skill do?

Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.

How do I install the Crw Map skill?

Claude Code plugin: /plugin marketplace add fastcrw/crw. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Crw Map popular?

It lives in fastcrw/crw, which has 1,268 GitHub stars and 102 forks, and is included in 1 curated skill list.

Related skills

Crw

1.3k

fastcrw/crw

Scrape, crawl, map, and search the web using fastCRW's native /v1 API. Use when the user needs web page content, site-wide extraction, URL discovery, or web search results. Single binary, 14 MB RAM; /v2 exists separately for Firecrawl migration.

Writing & ContentIn 1 list102

Crw

1.3k

fastcrw/crw

Scrape, crawl, map, search, parse, and extract web data with fastCRW — the open-source, self-hostable Firecrawl alternative (single Rust binary, ~14 MB RAM, Firecrawl-compatible /v1 + /v2 API). Use whenever the user needs page content, site-wide extraction, URL discovery, web search, PDF parsing, structured JSON from pages, or change tracking. Also use when the user mentions Firecrawl, Tavily, Crawl4AI, or "scrape/crawl/map/fetch/get the page/read this site/search the web" — crw is a drop-in for the Firecrawl SDKs.

Data & AnalyticsIn 1 list102

Crw Best Practices

1.3k

fastcrw/crw

Reference skill for building production-ready crw integrations. Covers verb selection, call surfaces (CLI/MCP/REST), post-filtering strategies, context-window hygiene, Hybrid RAG patterns, common pitfalls, and crw-specific operational considerations (search backend limits, renderer pool, proxy rotation). Load this when writing application code that embeds crw, designing a multi-step agent workflow, or debugging an integration that isn't behaving as expected.

AI, LLMs & AgentsIn 1 list102

Crw Scrape

1.3k

fastcrw/crw

Scrape a single known URL into clean markdown / HTML / links / structured JSON with fastCRW. Use when you already have the URL and want the page content — "scrape", "grab", "fetch", "pull", "read this page", "get the content of". Handles JavaScript-rendered SPAs automatically. Step 2 of the crw workflow ladder.

Data & AnalyticsIn 1 list102

Crw Dynamic Search

1.3k

fastcrw/crw

Programmatic web search and scrape with context isolation. Use for any research task where you need to search the web, filter results, and extract specific information — without flooding your context window with raw HTML and boilerplate. This is the single biggest token-saver in the crw skill set. Triggered by "search for", "look up", "find", "research", "what's the latest on", or any query that requires current web information. Also use when asked to "search and filter", "find the important parts", or any task where you suspect the raw output will be large (multi-page scrapes, news aggregatio

Data & AnalyticsIn 1 list102

Crw Watch

1.3k

fastcrw/crw

Detect what changed between two page snapshots with fastCRW — stateless diff as a REST primitive. Use when you need to track content changes, monitor a page for updates, or build a cron-based alert system: "has this page changed?", "alert me when pricing changes", "diff this week's scrape against last week's". Step 7 of the crw workflow ladder.

Data & AnalyticsIn 1 list102

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.