Intelligent Web Scraper

Self-learning intelligent web scraper agent - automatically analyzes page structure, handles pagination, anti-blocking, and discovers article series. No user configuration needed - AI decides everything.

65 stars 6 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add Yrzhe/claude-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add Yrzhe/claude-skills --skill intelligent-web-scraper
Manual install
git clone --depth 1 https://github.com/Yrzhe/claude-skills.git /tmp/claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills/plugins/intelligent-web-scraper/skills/intelligent-web-scraper ~/.claude/skills/intelligent-web-scraper

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Intelligent Web Scraper Agent

A self-aware, self-learning web scraping agent that accumulates experience over time.

## Prerequisites

> **IMPORTANT**: Before using this skill, ensure these dependencies are installed.

### One-Click Setup (Recommended)

```bash
# Navigate to skill directory and run setup
cd ~/.claude/skills/intelligent-web-scraper
chmod +x setup.sh && ./setup.sh

# Or for VPS/headless environment:
./setup.sh --headless
```

The setup script will:
- Detect your environment (macOS/Linux, GUI/headless)
- Install all Python dependencies
- Install Playwright browsers
- Set up Crawl4AI
- Create necessary directories

### Check Dependencies

```bash
# Check if everything is installed correctly
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py

# Auto-install missing dependencies
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --install

# Output as JSON (for programmatic use)
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --json
```

### Manual Setup (Alternative)

If you prefer manual installation:

```bash
# 1. Install Python dependencies
pip install -r ~/.claude/skills/intelligent-web-scraper/requirements.txt

# 2. Install Playwright browser
python -m playwright install chromium

# 3. Set up Crawl4AI
crawl4ai-setup

# 4. For VPS/headless Linux, also run:
python -m playwright install-deps chromium
```

### Required Tools

| Tool | Purpose | Installation |
|------|---------|--------------|
Read the full skill on GitHub

Details

Skill name
intelligent-web-scraper
Path
plugins/intelligent-web-scraper/skills/intelligent-web-scraper/SKILL.md
License
MIT
Repo updated
27 October 2026

FAQ

What does the Intelligent Web Scraper skill do?

Self-learning intelligent web scraper agent - automatically analyzes page structure, handles pagination, anti-blocking, and discovers article series. No user configuration needed - AI decides everything.

How do I install the Intelligent Web Scraper skill?

Claude Code plugin: /plugin marketplace add Yrzhe/claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Intelligent Web Scraper popular?

It lives in Yrzhe/claude-skills, which has 65 GitHub stars and 6 forks.

Related skills

Adaptive Compaction

65

Yrzhe/claude-skills

Adaptive add-on policy and recovery layer that decides WHEN to compact, prune, snapshot, or fork -- replacing fixed-percent auto-compaction across Claude Code, Codex, and MCP-capable hosts. Trigger on auto-compact timing or damage: "when should I compact", "is it safe to compact now or start a fresh session", "auto-compact fires too early/mid-task", "switching to an unrelated task but the window still has space", "context rot", "answers get worse the longer the session runs", "the agent forgot the plan or my decisions after it summarized", "add a layer on top that manages context without chang

AI, LLMs & Agents

Agentlog

65

Yrzhe/claude-skills

Load when the user wants to view, sync, or analyze multi-agent activity across multiple devices — Claude Code / Codex / Cursor / Maestri / browser-use sessions captured to a shared GitHub-synced pool, OR wants a distilled per-project context document so a fresh agent in a different tool can pick up where another left off. Triggers on "agentlog X" commands, "what did I do today across all my agents", "show me my pool", "sync the pool", "拉一下另一台机器上的 agent 记录", "看看 vps 上跑的", "跨设备 agent log", "跨工具上下文", "switch from Codex to Claude Code", "AGENTS.md / CLAUDE.md", "项目状态文档", "agent context". Do NOT lo

AI, LLMs & Agents

Akshare a Shares

65

Yrzhe/claude-skills

针对中国A股的行情与数据查询(基于本地 akshare 源代码与API映射)。当用户请求获取A股实时行情、历史K线(日/周/月)、分钟级分时(1/5/15/30/60)、复权数据(前复权/后复权)、股东户数(含详情)、以及巨潮资讯公告披露结果时使用本Skill。默认优先使用东方财富(EM)接口;如用户指定新浪(Sina)来源或需要特定接口,则按映射调用对应函数。可使用内置脚本输出CSV/JSON,或内联Python调用。

Software Development

Magic Mouth

113

Hmbown/Wizards-of-the-Ghosts

Magic Mouth is trigger → message. The entire craft is specifying the trigger boundary, the message payload, and the suppression/escalation rules. It is NOT: - One-time messages: "Send a Slack message now saying X" (no trigger, no automation) - Full chatbots: "Build a conversational AI that understands context and handles open-ended questions" (requires NLU, dialogue management) - Monitoring dashboards: "Set up Grafana with real-time graphs and trend analysis" (visualization, not message routing) - Silent traps/wards: "Revert changes silently and log who tried" (defensive code manipulation, not

Data & Analytics11

Healing Word

113

Hmbown/Wizards-of-the-Ghosts

In D&D, Healing Word heals at range with a bonus action — fast, remote, and efficient but less powerful than Cure Wounds. The real-world version is the remote quick fix: a one-liner command, a config change pushed through a dashboard, a quick Slack instruction to the person on-call, or an automated rollback trigger. You do not need to be deep in the system to apply this fix.

Data & Analytics11

Senior Data Analyst

65

Yrzhe/claude-skills

Use when the user wants a real data analysis on a dataset — finding what drives a metric, testing whether a difference is real, building a predictive model, reducing many variables to a few, or interpreting what numbers mean for the business. Trigger phrases include "analyze this data", "what drives X", "is this significant", "build a model to predict", "find the factors", "do a proper analysis", "what does this data mean". Skip when the user only wants a chart, only wants raw data pulled with no question, asks a closed factual question with no data, or just wants APA-formatted stats output.

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.