Headless Web Scraping
Extract data from web pages using the scrapling Python library — select the appropriate fetcher tier (HTTP, stealth Chromium, or full browser automation) based on target site defenses, configure headless browsing, and extract structured data with CSS selectors. Use when WebFetch is insufficient for JS-rendered pages, anti-bot-protected sites, or structured multi-element extraction requiring DOM traversal.
Install
npx skills add pjt222/agent-almanac --skill headless-web-scrapinggit clone --depth 1 https://github.com/pjt222/agent-almanac.git /tmp/agent-almanac
mkdir -p ~/.claude/skills
cp -r /tmp/agent-almanac/i18n/caveman-lite/skills/headless-web-scraping ~/.claude/skills/headless-web-scrapingPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Headless Web Scraping Extract data from web pages that resist simple HTTP requests — JS-rendered content, Cloudflare-protected sites, and dynamic SPAs — using scrapling's three-tier fetcher architecture and CSS-based data extraction. ## When to Use - Target page requires JavaScript rendering (SPA, React, Vue) - Site has anti-bot protections (Cloudflare Turnstile, TLS fingerprinting) - You need structured extraction of multiple elements via CSS selectors - Simple `WebFetch` or `requests.get()` returns empty or blocked responses - Extracting tabular data, link lists, or repeated DOM structures at scale ## Inputs - **Required**: Target URL or list of URLs to scrape - **Required**: Data to extract (CSS selectors, field names, or description of target elements) - **Optional**: Fetcher tier override (default: auto-select based on site behavior) - **Optional**: Output format (default: JSON; alternatives: CSV, Python dict) - **Optional**: Rate limit delay in seconds (default: 1) ## Procedure ### Step 1: Select Fetcher Tier Determine which scrapling fetcher matches the target site's defenses.
Details
- Skill name
headless-web-scraping- Repository
- pjt222/agent-almanac
- Path
i18n/caveman-lite/skills/headless-web-scraping/SKILL.md- License
- MIT
- Repo updated
- 18 November 2026
- Listed in
- BehiSecc/awesome-claude-skills
FAQ
What does the Headless Web Scraping skill do?
Extract data from web pages using the scrapling Python library — select the appropriate fetcher tier (HTTP, stealth Chromium, or full browser automation) based on target site defenses, configure headless browsing, and extract structured data with CSS selectors. Use when WebFetch is insufficient for JS-rendered pages, anti-bot-protected sites, or structured multi-element extraction requiring DOM traversal.
How do I install the Headless Web Scraping skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add pjt222/agent-almanac --skill headless-web-scraping. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Headless Web Scraping popular?
It lives in pjt222/agent-almanac, which has 45 GitHub stars and 4 forks, and is included in 1 curated skill list.
Related skills
Adapt Architecture
45pjt222/agent-almanac
Execute structural metamorphosis using strangler fig migration, chrysalis phases, and interface preservation. Covers transformation planning, parallel running, progressive cutover, rollback design, and post-metamorphosis stabilization for system architecture evolution. Use when assess-form has classified the system as READY for transformation, when migrating from monolith to microservices, when replacing a core subsystem while dependents continue operating, or when any architectural change must be gradual rather than big-bang.
Adaptic
45pjt222/agent-almanac
Master skill composing the 5-step synoptic cycle for panoramic synthesis across multiple domains. Orchestrates meditate, expand-awareness, observe, awareness, integrate-gestalt, and express-insight into a coherent process that produces unified understanding rather than sequential compromise. Use when a problem genuinely spans 3+ domains and the interactions between domains matter more than depth in any one, when sequential analysis feels like compromise rather than integration, or before major architectural decisions affecting multiple stakeholders.
Add Puzzle Type
45pjt222/agent-almanac
Scaffold a new puzzle type across all 10+ pipeline integration points in jigsawR. Creates the core puzzle module, wires it into the unified pipeline (generation, positioning, rendering, adjacency), adds ggpuzzle geom/stat layers, updates DESCRIPTION and config.yml, extends the Shiny app, and creates a comprehensive test suite. Use when adding a completely new puzzle type to the package or following the 10-point integration checklist to ensure nothing is missed end-to-end.
Heal Guidance
45pjt222/agent-almanac
Guide a person through healing modalities including energy work (reiki, chakra balancing), herbal remedies, basic first aid stabilization, and holistic techniques (breathwork, visualization, body scan). AI coaches the practitioner through assessment triage, modality selection, energetic connection, remedy preparation, and integration. Use when a person describes a physical ailment or injury, reports energetic imbalance (fatigue, emotional stagnation), wants coaching through a holistic breathwork or visualization session, or needs post-meditation integration with directed healing attention.
Generate Status Report
45pjt222/agent-almanac
Generate a project status report by reading existing artifacts (charter, backlog, sprint plan, WBS), calculating metrics, identifying blockers, and summarizing progress with RAG indicators for schedule, scope, budget, and quality. Use at the end of a sprint or reporting period, when stakeholders request a health update, before steering committee or governance meetings, or when a new blocker or risk materializes mid-project.
Manage Tcg Collection
45pjt222/agent-almanac
Organize, track, and value a trading card game collection. Covers inventory methods, storage best practices, grade-based valuation, want-list management, and collection analytics for Pokemon, MTG, Flesh and Blood, and Kayou cards. Use when starting a new collection and setting up inventory tracking, cataloging an existing collection that has grown beyond casual knowledge, valuing a collection for insurance or sale, or deciding which cards to submit for professional grading based on value potential.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.