Web Scraper
Scrape web pages and save as HTML or Markdown (with text and images). Minimal dependencies - only requests and beautifulsoup4. Use when the user provides a URL and wants to download/archive the content locally.
Install
npx skills add agentbay-ai/agentbay-skills --skill web-scrapergit clone --depth 1 https://github.com/agentbay-ai/agentbay-skills.git /tmp/agentbay-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agentbay-skills/web-scraper ~/.claude/skills/web-scraperPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Web Scraper
Fetch web page content (text + images) and save as HTML or Markdown locally.
**Minimal dependencies**: Only requires `requests` and `beautifulsoup4` - no browser automation.
**Default behavior**: Downloads images to local `images/` directory automatically.
## Quick start
### Single page
```bash
{baseDir}/scripts/scrape.py --url "https://example.com" --format html --output /tmp/page.html
{baseDir}/scripts/scrape.py --url "https://example.com" --format md --output /tmp/page.md
```
### Recursive (follow links)
```bash
{baseDir}/scripts/scrape.py --url "https://docs.example.com" --format md --recursive --max-depth 2 --output ~/Downloads/docs-archive
```
## Setup
Requires Python 3.8+ and minimal dependencies:
```bash
cd {baseDir}
pip install -r requirements.txt
```
Or install manually:
```bash
pip install requests beautifulsoup4
```
**Note**: No browser or driver needed - uses pure HTTP requests.
## Inputs to collect
### Single page mode
- **URL**: The web page to scrape (required)
- **Format**: `html` or `md` (default: `html`)
- **Output path**: Where to save the file (default: current directory with auto-generated name)
- **Images**: Downloads images by default (use `--no-download-images` to disable)
### Recursive mode (--recursive)
- **URL**: Starting point for recursive scraping
- **Format**: `html` or `md`
- **Output directory**: Where to save all scraped pages
- **Max depth**: How many levels deep to follow links (default: 2)Details
- Skill name
web-scraper- Repository
- agentbay-ai/agentbay-skills
- Path
web-scraper/SKILL.md- Repo updated
- 12 February 2026
- Listed in
- BehiSecc/awesome-claude-skills
FAQ
What does the Web Scraper skill do?
Scrape web pages and save as HTML or Markdown (with text and images). Minimal dependencies - only requests and beautifulsoup4. Use when the user provides a URL and wants to download/archive the content locally.
How do I install the Web Scraper skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add agentbay-ai/agentbay-skills --skill web-scraper. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Web Scraper popular?
It lives in agentbay-ai/agentbay-skills, which has 51 GitHub stars and 5 forks, and is included in 1 curated skill list.
Related skills
Agentbay Aio Skills
51agentbay-ai/agentbay-skills
用于任何「运行/执行/评估代码」的请求(如 run this code、execute script、用 Python 画图、跑代码/画图/执行脚本)。通过 AgentBay SDK 创建 code_latest 沙箱,调用 run_code 执行并返回结果。支持 Python、JavaScript、R、Java。
Agentbay Monitor Skills
51agentbay-ai/agentbay-skills
舆情监控技能,最终产出舆情报告。当用户问「某事件/话题舆情如何」「舆论怎么样」「做舆情分析」「运行舆情分析」或按关键词/平台爬取并生成舆情报告时,使用本技能。约定:凡舆情相关意图即执行全流程(爬取→情感分析→生成报告)。爬取由本技能完成;情感分析由主 Agent 按提示词自主判断;报告由 generate_report 生成。
Amap Traffic
51agentbay-ai/agentbay-skills
高德地图实时路况查询与最优自驾路线规划技能。基于高德交通态势API和路径规划API,提供实时拥堵信息和最快路线建议。
Tmux
44iamclancyliang/nanobot-eino
Remote-control tmux sessions for interactive CLIs by sending keystrokes and scraping pane output.
Alterlab Vaex
87AlterLab-IEU/AlterLab-Academic-Skills
Out-of-core tabular analytics with Vaex — memory-mapped HDF5/Arrow/Parquet via vaex.open, lazy virtual columns, delayed single-pass aggregations on billion-row tables, binned histograms/heatmaps, and vaex.ml transformers on one machine. Vaex is in minimal-maintenance mode (vaex-core 4.19 from Sep 2025 supports only Python 3.9–3.12 and pandas < 3), so use it to run, fix, or migrate existing Vaex code; for new larger-than-RAM work prefer Polars lazy/streaming queries (alterlab-polars) or Dask (alterlab-dask). Use when a project already depends on Vaex, when opening memory-mapped HDF5/Arrow files
1p Tagging Pixels
79LeadMagic/gtm-skills
Set up first-party tracking and analytics — website pixels, conversion tracking, UTM architecture, cookie consent, server-side tagging vs client-side, 1P vs 3P data strategy, and identity resolution. Use when setting up tracking, implementing pixels, building attribution, or preparing for privacy-first measurement. Triggers on: "pixel", "tracking", "conversion tracking", "first-party data", "server-side tagging", "UTM", "cookie consent", "1P vs 3P", "identity resolution", or any tracking implementation request.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.