Intelligent Web Scraper
Self-learning intelligent web scraper agent - automatically analyzes page structure, handles pagination, anti-blocking, and discovers article series. No user configuration needed - AI decides everything.
Install
/plugin marketplace add Yrzhe/claude-skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add Yrzhe/claude-skills --skill intelligent-web-scrapergit clone --depth 1 https://github.com/Yrzhe/claude-skills.git /tmp/claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills/plugins/intelligent-web-scraper/skills/intelligent-web-scraper ~/.claude/skills/intelligent-web-scraperPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Intelligent Web Scraper Agent A self-aware, self-learning web scraping agent that accumulates experience over time. ## Prerequisites > **IMPORTANT**: Before using this skill, ensure these dependencies are installed. ### One-Click Setup (Recommended) ```bash # Navigate to skill directory and run setup cd ~/.claude/skills/intelligent-web-scraper chmod +x setup.sh && ./setup.sh # Or for VPS/headless environment: ./setup.sh --headless ``` The setup script will: - Detect your environment (macOS/Linux, GUI/headless) - Install all Python dependencies - Install Playwright browsers - Set up Crawl4AI - Create necessary directories ### Check Dependencies ```bash # Check if everything is installed correctly python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py # Auto-install missing dependencies python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --install # Output as JSON (for programmatic use) python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --json ``` ### Manual Setup (Alternative) If you prefer manual installation: ```bash # 1. Install Python dependencies pip install -r ~/.claude/skills/intelligent-web-scraper/requirements.txt # 2. Install Playwright browser python -m playwright install chromium # 3. Set up Crawl4AI crawl4ai-setup # 4. For VPS/headless Linux, also run: python -m playwright install-deps chromium ``` ### Required Tools | Tool | Purpose | Installation | |------|---------|--------------|
Details
- Skill name
intelligent-web-scraper- Repository
- Yrzhe/claude-skills
- Path
plugins/intelligent-web-scraper/skills/intelligent-web-scraper/SKILL.md- License
- MIT
- Repo updated
- 27 October 2026
FAQ
What does the Intelligent Web Scraper skill do?
Self-learning intelligent web scraper agent - automatically analyzes page structure, handles pagination, anti-blocking, and discovers article series. No user configuration needed - AI decides everything.
How do I install the Intelligent Web Scraper skill?
Claude Code plugin: /plugin marketplace add Yrzhe/claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Intelligent Web Scraper popular?
It lives in Yrzhe/claude-skills, which has 65 GitHub stars and 6 forks.
Related skills
Adaptive Compaction
65Yrzhe/claude-skills
Adaptive add-on policy and recovery layer that decides WHEN to compact, prune, snapshot, or fork -- replacing fixed-percent auto-compaction across Claude Code, Codex, and MCP-capable hosts. Trigger on auto-compact timing or damage: "when should I compact", "is it safe to compact now or start a fresh session", "auto-compact fires too early/mid-task", "switching to an unrelated task but the window still has space", "context rot", "answers get worse the longer the session runs", "the agent forgot the plan or my decisions after it summarized", "add a layer on top that manages context without chang
Agentlog
65Yrzhe/claude-skills
Load when the user wants to view, sync, or analyze multi-agent activity across multiple devices — Claude Code / Codex / Cursor / Maestri / browser-use sessions captured to a shared GitHub-synced pool, OR wants a distilled per-project context document so a fresh agent in a different tool can pick up where another left off. Triggers on "agentlog X" commands, "what did I do today across all my agents", "show me my pool", "sync the pool", "拉一下另一台机器上的 agent 记录", "看看 vps 上跑的", "跨设备 agent log", "跨工具上下文", "switch from Codex to Claude Code", "AGENTS.md / CLAUDE.md", "项目状态文档", "agent context". Do NOT lo
Akshare a Shares
65Yrzhe/claude-skills
针对中国A股的行情与数据查询(基于本地 akshare 源代码与API映射)。当用户请求获取A股实时行情、历史K线(日/周/月)、分钟级分时(1/5/15/30/60)、复权数据(前复权/后复权)、股东户数(含详情)、以及巨潮资讯公告披露结果时使用本Skill。默认优先使用东方财富(EM)接口;如用户指定新浪(Sina)来源或需要特定接口,则按映射调用对应函数。可使用内置脚本输出CSV/JSON,或内联Python调用。
Magic Mouth
113Hmbown/Wizards-of-the-Ghosts
Magic Mouth is trigger → message. The entire craft is specifying the trigger boundary, the message payload, and the suppression/escalation rules. It is NOT: - One-time messages: "Send a Slack message now saying X" (no trigger, no automation) - Full chatbots: "Build a conversational AI that understands context and handles open-ended questions" (requires NLU, dialogue management) - Monitoring dashboards: "Set up Grafana with real-time graphs and trend analysis" (visualization, not message routing) - Silent traps/wards: "Revert changes silently and log who tried" (defensive code manipulation, not
Healing Word
113Hmbown/Wizards-of-the-Ghosts
In D&D, Healing Word heals at range with a bonus action — fast, remote, and efficient but less powerful than Cure Wounds. The real-world version is the remote quick fix: a one-liner command, a config change pushed through a dashboard, a quick Slack instruction to the person on-call, or an automated rollback trigger. You do not need to be deep in the system to apply this fix.
Senior Data Analyst
65Yrzhe/claude-skills
Use when the user wants a real data analysis on a dataset — finding what drives a metric, testing whether a difference is real, building a predictive model, reducing many variables to a few, or interpreting what numbers mean for the business. Trigger phrases include "analyze this data", "what drives X", "is this significant", "build a model to predict", "find the factors", "do a proper analysis", "what does this data mean". Skip when the user only wants a chart, only wants raw data pulled with no question, asks a closed factual question with no data, or just wants APA-formatted stats output.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.