Web Scraper

Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes

150 stars 32 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add guia-matthieu/clawfu-skills --skill web-scraper
Manual install
git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git /tmp/clawfu-skills
mkdir -p ~/.claude/skills
cp -r /tmp/clawfu-skills/skills/automation/web-scraper ~/.claude/skills/web-scraper

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Web Scraper

> Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.

## When to Use This Skill

- **Competitor research** - Scrape pricing, features, positioning
- **Lead generation** - Extract contact info from directories
- **Content audit** - Pull headings, links, meta data
- **Price monitoring** - Track competitor pricing changes
- **Data collection** - Gather research data from multiple sources

## What Claude Does vs What You Decide

| Claude Does | You Decide |
|-------------|------------|
| Structures analysis frameworks | Strategic priorities |
| Synthesizes market data | Competitive positioning |
| Identifies opportunities | Resource allocation |
| Creates strategic options | Final strategy selection |
| Suggests implementation approaches | Execution decisions |

## Dependencies

```bash
pip install beautifulsoup4 requests pandas click lxml
```

## Commands

### Scrape Elements
```bash
python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"
```

### Extract Links
```bash
python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only
```

### Extract Emails
```bash
python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2
```

### Extract Structured Data
Read the full skill on GitHub

Details

Skill name
web-scraper
Path
skills/automation/web-scraper/SKILL.md
License
MIT
Repo updated
1 October 2026

FAQ

What does the Web Scraper skill do?

Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes

How do I install the Web Scraper skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add guia-matthieu/clawfu-skills --skill web-scraper. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Web Scraper popular?

It lives in guia-matthieu/clawfu-skills, which has 150 GitHub stars and 32 forks, and is included in 1 curated skill list.

Related skills

Aarrr Metrics

150

guia-matthieu/clawfu-skills

Measure and optimize growth using the AARRR (Pirate Metrics) framework with stage-specific KPIs and funnel analysis

Marketing, Sales & BusinessIn 1 list32

Ab Test Stats

150

guia-matthieu/clawfu-skills

Calculate A/B test statistical significance. Use when: determining if test results are significant; calculating required sample size; estimating test duration; analyzing conversion experiments; making data-driven decisions

Testing & QAIn 1 list32

Account Health

150

guia-matthieu/clawfu-skills

Assess customer account health using product usage, support sentiment, payment status, and relationship signals

Finance & CryptoIn 1 list32

Data Visualization

223

NeverSight/learn-skills.dev

Data visualization with chart selection, color theory, and annotation best practices. Covers chart types (bar, line, scatter, heatmap), axes rules, and storytelling with data. Use for: charts, graphs, dashboards, reports, presentations, infographics, data stories. Triggers: data visualization, chart, graph, data chart, bar chart, line chart, scatter plot, data viz, visualization, dashboard chart, infographic data, data presentation, chart design, plot, heatmap, pie chart alternative

Data & Analytics42

Twitter X Scraping

201

swyxio/skills

Use when scraping public Twitter/X timelines or lists through Nitter-style mirrors with axios and Playwright fallback, including anti-bot challenge handling and pagination limits.

Data & Analytics12

Fusion Data Analysis

223

NeverSight/learn-skills.dev

Fusion plasma data analysis across the tokamak/stellarator research pipeline. Use when the user asks to analyze, interpret, or QA equilibrium reconstructions (EFIT, LIUQE, G-EQDSK, gfile, eqdsk, flux-surface geometry, safety factor q-profile, magnetic shear s, normalized flux ψ_N, ρ_tor, r/a, elongation κ, triangularity δ), kinetic profiles (electron/ion temperature T_e/T_i, electron density n_e, toroidal/impurity rotation v_φ/v_t, profile fitting with polynomial/tanh/mtanh/Gaussian, normalized gradient scale lengths R/L_T and R/L_n, profile alignment and interpolation, error propagation), tra

Data & Analytics42

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.