Web Scraper

Scrape web pages and save as HTML or Markdown (with text and images). Minimal dependencies - only requests and beautifulsoup4. Use when the user provides a URL and wants to download/archive the content locally.

51 stars 5 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add agentbay-ai/agentbay-skills --skill web-scraper
Manual install
git clone --depth 1 https://github.com/agentbay-ai/agentbay-skills.git /tmp/agentbay-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agentbay-skills/web-scraper ~/.claude/skills/web-scraper

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Web Scraper

Fetch web page content (text + images) and save as HTML or Markdown locally.

**Minimal dependencies**: Only requires `requests` and `beautifulsoup4` - no browser automation.

**Default behavior**: Downloads images to local `images/` directory automatically.

## Quick start

### Single page

```bash
{baseDir}/scripts/scrape.py --url "https://example.com" --format html --output /tmp/page.html
{baseDir}/scripts/scrape.py --url "https://example.com" --format md --output /tmp/page.md
```

### Recursive (follow links)

```bash
{baseDir}/scripts/scrape.py --url "https://docs.example.com" --format md --recursive --max-depth 2 --output ~/Downloads/docs-archive
```

## Setup

Requires Python 3.8+ and minimal dependencies:

```bash
cd {baseDir}
pip install -r requirements.txt
```

Or install manually:

```bash
pip install requests beautifulsoup4
```

**Note**: No browser or driver needed - uses pure HTTP requests.

## Inputs to collect

### Single page mode

- **URL**: The web page to scrape (required)
- **Format**: `html` or `md` (default: `html`)
- **Output path**: Where to save the file (default: current directory with auto-generated name)
- **Images**: Downloads images by default (use `--no-download-images` to disable)

### Recursive mode (--recursive)

- **URL**: Starting point for recursive scraping
- **Format**: `html` or `md`
- **Output directory**: Where to save all scraped pages
- **Max depth**: How many levels deep to follow links (default: 2)
Read the full skill on GitHub

Details

Skill name
web-scraper
Path
web-scraper/SKILL.md
Repo updated
12 February 2026

FAQ

What does the Web Scraper skill do?

Scrape web pages and save as HTML or Markdown (with text and images). Minimal dependencies - only requests and beautifulsoup4. Use when the user provides a URL and wants to download/archive the content locally.

How do I install the Web Scraper skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add agentbay-ai/agentbay-skills --skill web-scraper. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Web Scraper popular?

It lives in agentbay-ai/agentbay-skills, which has 51 GitHub stars and 5 forks, and is included in 1 curated skill list.

Related skills

Agentbay Aio Skills

51

agentbay-ai/agentbay-skills

用于任何「运行/执行/评估代码」的请求(如 run this code、execute script、用 Python 画图、跑代码/画图/执行脚本)。通过 AgentBay SDK 创建 code_latest 沙箱,调用 run_code 执行并返回结果。支持 Python、JavaScript、R、Java。

AI, LLMs & AgentsIn 1 list

Agentbay Monitor Skills

51

agentbay-ai/agentbay-skills

舆情监控技能,最终产出舆情报告。当用户问「某事件/话题舆情如何」「舆论怎么样」「做舆情分析」「运行舆情分析」或按关键词/平台爬取并生成舆情报告时,使用本技能。约定:凡舆情相关意图即执行全流程(爬取→情感分析→生成报告)。爬取由本技能完成;情感分析由主 Agent 按提示词自主判断;报告由 generate_report 生成。

AI, LLMs & AgentsIn 1 list

Amap Traffic

51

agentbay-ai/agentbay-skills

高德地图实时路况查询与最优自驾路线规划技能。基于高德交通态势API和路径规划API,提供实时拥堵信息和最快路线建议。

Software DevelopmentIn 1 list

Tmux

44

iamclancyliang/nanobot-eino

Remote-control tmux sessions for interactive CLIs by sending keystrokes and scraping pane output.

Data & Analytics

Alterlab Vaex

87

AlterLab-IEU/AlterLab-Academic-Skills

Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-core 4.19 from Sep 2025 supports only Python 3.9–3.12 and pandas < 3), so use it to run, fix, or migrate existing Vaex code; for new larger-than-RAM work prefer Polars lazy/streaming queries (alterlab-polars) or Dask (alterlab-dask). Use when a project already depends on Vaex, when opening memory-mapped HDF5/Arrow files

Data & AnalyticsIn 1 list18

1p Tagging Pixels

79

LeadMagic/gtm-skills

Set up first-party tracking and analytics — website pixels, conversion tracking, UTM architecture, cookie consent, server-side tagging vs client-side, 1P vs 3P data strategy, and identity resolution. Use when setting up tracking, implementing pixels, building attribution, or preparing for privacy-first measurement. Triggers on: "pixel", "tracking", "conversion tracking", "first-party data", "server-side tagging", "UTM", "cookie consent", "1P vs 3P", "identity resolution", or any tracking implementation request.

Data & AnalyticsIn 1 list18

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.