Web Scraper

Crawl and extract structured data from websites. Uses a tiered approach -- Zo built-ins for simple fetches, BeautifulSoup for custom parsing, Crawl4AI for deep crawls and LLM-powered extraction. Use when you need to scrape a website, extract structured data, or crawl multiple pages.

46 stars 56 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add zocomputer/skills --skill web-scraper
Manual install
git clone --depth 1 https://github.com/zocomputer/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/Community/web-scraper ~/.claude/skills/web-scraper

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

## CLI

```bash
python3 /home/workspace/Skills/web-scraper/scripts/scraper.py <command> [options]
```

| Command | Description |
|---------|-------------|
| `scrape <url>` | Fetch page content as markdown |
| `scrape <url> --css <selector>` | Extract elements matching a CSS selector |
| `scrape <url> --schema <json-file>` | Extract structured data using a JSON CSS schema |
| `scrape <url> --llm-extract "<instruction>"` | Use LLM to extract structured data (Crawl4AI) |
| `crawl <url> --depth N --max-pages N` | Deep crawl following links (BFS) |
| `batch <file> [--css <selector>]` | Scrape multiple URLs from a file (one per line) |

## Options

| Flag | Description |
|------|-------------|
| `--output <file>` | Save output to file instead of stdout |
| `--format json\|md\|text` | Output format (default: md for scrape, json for extract) |
| `--js` | Force JavaScript rendering via Playwright |
| `--wait <seconds>` | Wait after page load before extracting |
| `--timeout <seconds>` | Request timeout (default: 30) |

## Tiered Approach

The scraper automatically picks the lightest tool that can do the job:

1. **Simple fetch** (`scrape <url>`) -- Uses httpx + BeautifulSoup. Fast, no browser overhead.
2. **CSS extraction** (`--css` or `--schema`) -- BeautifulSoup with selectors. Still fast.
3. **JS rendering** (`--js` flag, or auto-fallback) -- Playwright via Crawl4AI. For SPAs and dynamic content.
4. **Deep crawl** (`crawl` command) -- Crawl4AI with BFS link following.
Read the full skill on GitHub

Details

Skill name
web-scraper
Path
Community/web-scraper/SKILL.md
License
MIT
Repo updated
29 July 2026

FAQ

What does the Web Scraper skill do?

Crawl and extract structured data from websites. Uses a tiered approach -- Zo built-ins for simple fetches, BeautifulSoup for custom parsing, Crawl4AI for deep crawls and LLM-powered extraction. Use when you need to scrape a website, extract structured data, or crawl multiple pages.

How do I install the Web Scraper skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add zocomputer/skills --skill web-scraper. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Web Scraper popular?

It lives in zocomputer/skills, which has 46 GitHub stars and 56 forks.

Related skills

Add Flashcards

46

zocomputer/skills

Converts webpages, articles, or documents into study flashcards that you can review with spaced repetition

Writing & Content56

Blog Site

46

zocomputer/skills

Build a blog or newsletter to share your thoughts.

Writing & Content56

Blogwatcher

46

zocomputer/skills

Monitor blogs and RSS/Atom feeds for updates using the blogwatcher CLI.

Writing & Content56

09 Consistency Validation

44

ChenXiangYpily1234/paper-storycraft-skills

Use for independent full-paper or pre-submission audits of claims, references, formulas, statistics, cross-references, figures and document integrity. Review only unless edits are explicitly requested.

Data & Analytics

Status

40

ArogyaReddy/alirezarezvani-claude-skills

Memory health dashboard showing line counts, topic files, capacity, stale entries, and recommendations.

Data & Analytics11

So What Generator

33

AnugamChakra/think-like-a-strategy-consultant

Turns an observation, data point, chart, or finding into a real insight — the implication and the action — instead of just restating what happened. Use this skill whenever someone shares a fact, metric, chart, or table and needs to know what it means, asks "so what", "what's the takeaway", "why does this matter", "what should we do about this", or is presenting findings that read as observations rather than conclusions. Trigger it whenever a user has data but hasn't extracted the decision from it, even if they don't use the phrase "so what". This is the instinct that turns a background slide i

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.