Crawl4ai

Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.

49 stars 13 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)

npx skills add brettdavies/crawl4ai-skill --skill crawl4ai

Manual install

git clone --depth 1 https://github.com/brettdavies/crawl4ai-skill.git /tmp/crawl4ai-skill
mkdir -p ~/.claude/skills
cp -r /tmp/crawl4ai-skill/. ~/.claude/skills/crawl4ai

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Crawl4AI

**Verified against `crawl4ai`** [`VERSION`](VERSION). PEP 723 pins in `scripts/*.py` and `tests/*.py` floor at that
version.

## Overview

Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't
reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable
schema-based extraction.

This skill exposes both interfaces of the underlying library:

- **CLI** (`crwl`) — quick, scriptable commands: [CLI Guide](references/cli-guide.md)
- **Python SDK** — full programmatic control: [SDK Guide](references/sdk-guide.md)

## Invoked with a URL argument

When the user runs `/crawl4ai <url>` with a single URL and no further qualifier, treat it as the JS-heavy fetch case and
default to:

```bash
crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown
```

`wait_until=networkidle` waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't
named a specific element on a JS-rendered page. (Avoid `wait_for=css:body`: `<body>` exists at t=0 on every HTML
response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to
`wait_for=css:<selector>` if the user named a specific element. Skip the default and route to the relevant section below
for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each

Read the full skill on GitHub

Details

Skill name: crawl4ai
Repository: brettdavies/crawl4ai-skill
Repo updated: 5 October 2026

FAQ

What does the Crawl4ai skill do?

How do I install the Crawl4ai skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add brettdavies/crawl4ai-skill --skill crawl4ai. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Crawl4ai popular?

It lives in brettdavies/crawl4ai-skill, which has 49 GitHub stars and 13 forks.

Related skills

Web Analytics

SeoToolkit/web-analytics-agent-skill

Analyze web traffic, SEO performance, and search engine trends using Google Search Console, Google Analytics 4, and Bing Webmaster Tools. Use when the user asks to analyze traffic, check SEO, or find keyword opportunities.

Data & Analytics15

Anyviz

TseringYuu/anyviz

Use when 用户请求选择、创建、改进或统一数据可视化，并需按数据与意图选择图表、主题及 D3.js、ECharts、Mapbox、Three.js、Matplotlib、Plotly 或 ggplot2 技术栈。

Data & Analytics

ThinkWiki

wzdavid/ThinkWiki

Use this skill when the user wants to create, maintain, query, or visualize a local Markdown knowledge base. ThinkWiki can initialize a wiki, import files or webpages, collect inbox items, answer from existing knowledge, save higher-value outputs, and generate HTML viewer and graph artifacts.

Data & Analytics

Bys Personal Dashboard

qkgecn93/bys-personal-dashboard

不一书个人工作台生成器。通过一场引导式访谈，帮用户（包括完全不懂代码的人）做出一个属于自己的手机端个人工作台—— 左侧常驻功能栏、右侧卡片流，打开就是今日计划、打卡、记账、体重、日程、单词、树洞等自己每天真正要用的东西，可以部署上线并添加到手机桌面，像一个 App 一样用。当用户说"初始化不一书个人工作台生成器"、"初始化 bys-personal-dashboard"、"我想做一个自己的个人工作台"、 "抖音上那种个人工作台我也想要一个"、"把我的日常安排做成一个手机页面"时必须触发；即使用户只是丢来一张别人的工作台截图说"我也想要这种"，也应触发。已有工作台后说"给我的工作台加个XX功能"、"工作台换个配色"、"工作台改一下"时，走迭代流程而不是重新初始化。用户没说"工作台"这个词时，用三个特征判断：①在手机上用 ②每天都会打开 ③装的是用户自己的私人数据（体重、账目、打卡、待办、日记）。三个同时成立就该触发，比如"我想把体重记账待办整理到一个地方、手机上能点开"——这就是工作台，只是用户不知道这个叫法。三个缺任何一个都不要触发。功能只有一个也算——"我老忘记吃药，想每天在手机上勾一下"就是种子需求，别因为"听起来只是个小功能"就不触发。判据是数据属于谁，不是数据存在哪：用户自己的记账 Excel 想做成手机上每天看的页面，是工作台；公司销售数据做报表，不是。

Data & Analytics11

Supergis

OpenCesium/SuperGIS-Skills

Implement maps and geospatial visualization for OpenLayers, Cesium, Leaflet, MapLibre, Mapbox, maptalks, SuperMap, MapGIS, ArcGIS, GeoScene, AMap, Baidu Map, Tencent Map, Tianditu, KQGIS, NavInfo, Google Maps, Three.js, deck.gl, AntV L7, and MapV. Use when the user says supergis or names one of these libraries and asks for a map, layer, terrain, 3D Tiles, vector tile, GeoJSON, WMS, WMTS, draw, measure, query, or visualization. Open modules/<library>/index.md and the matching example, then adapt that example's full code to the host project.

Data & Analytics

Easyplot

110

Rimagination/easyplot

Use when the user asks for scientific data preparation, exploratory analysis, statistical inference, model evaluation, or research figures in R or Python, including analysis-only requests, multi-panel composition, and publication export. EasyPlot is the user's single entry point for research data analysis and visualization; exclude unrelated business dashboards and general application development.

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.