Crawlee

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

213 stars 34 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add TerminalSkills/skills --skill crawlee
Manual install
git clone --depth 1 https://github.com/TerminalSkills/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/skills/crawlee ~/.claude/skills/crawlee

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Crawlee

## Overview

Crawlee is a web scraping and crawling library that handles the hard parts — request queuing, retries, proxy rotation, browser fingerprinting, and rate limiting. Use Cheerio for fast HTML-only scraping or Playwright/Puppeteer for JavaScript-rendered pages. Built-in storage for datasets, request queues, and key-value stores. Scales from single pages to millions of URLs.

## When to Use

- Scraping data from websites (product prices, job listings, articles)
- Crawling entire sites for content or link analysis
- JavaScript-rendered pages (SPAs, React/Vue sites)
- Scraping at scale with proxy rotation and anti-blocking
- Structured data extraction with automatic retries

## Instructions

### Setup

```bash
npx crawlee create my-crawler        # official template, or add to an existing project:
npm install crawlee                  # CheerioCrawler / HttpCrawler only
npm install crawlee playwright       # browser crawling (Playwright or Puppeteer are separate installs)
npx playwright install chromium
```

Use ES modules (`"type": "module"` in package.json) so top-level `await crawler.run(...)` works. Checked against crawlee 3.18.

### HTTP Crawling with Routing (Fast, No Browser)

Give start requests a `label` and route them to handlers with a router; `enqueueLinks` stays on the same hostname by default and deduplicates URLs.
Read the full skill on GitHub

Details

Skill name
crawlee
Path
skills/crawlee/SKILL.md
License
Apache-2.0
Repo updated
8 November 2026

FAQ

What does the Crawlee skill do?

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

How do I install the Crawlee skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add TerminalSkills/skills --skill crawlee. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Crawlee popular?

It lives in TerminalSkills/skills, which has 213 GitHub stars and 34 forks.

Related skills

3dsmax Rendering

213

TerminalSkills/skills

Scripts and configures production rendering in Autodesk 3ds Max with the V-Ray and Corona renderers: output size and files, render elements, denoising, light mix, batch and command-line rendering, and network rendering. Use when a user asks to set up a production render, render several cameras in one batch, render from the command line or on a render farm, add render passes for compositing, or cut render time for archviz and product shots.

Other34

3dsmax Scripting

213

TerminalSkills/skills

Covers scripting Autodesk 3ds Max, the 3D modeling and rendering application, with MAXScript and Python (pymxs): scene manipulation, object creation, material assignment, camera and light setup, batch operations, and file I/O. Use when tasks involve automating repetitive 3ds Max workflows, batch processing scenes, running scripts headless with 3dsmaxbatch, creating custom tools, or scripting scene setup for archviz, product visualization, or VFX.

Productivity & Notes34

3proxy

213

TerminalSkills/skills

3proxy is a small open-source proxy server that runs HTTP/HTTPS, SOCKS4/5, SNI and TCP/UDP port-mapping proxies from one config file. Use when a user asks to set up an HTTP or SOCKS5 proxy, add proxy users and passwords, write 3proxy access rules, chain or rotate upstream (parent) proxies, limit bandwidth, connections or monthly traffic per user, run 3proxy in Docker, or fix a 3proxy.cfg that will not start.

DevOps & Cloud34

CSV Parse

213

TerminalSkills/skills

Parse and generate CSV files with the csv package — stream large files, handle custom delimiters, transform records, validate data, and generate CSV output from objects. Use when tasks involve data import/export, ETL pipelines, processing uploaded CSV files, or generating downloadable reports.

Data & Analytics34

Crawl4ai

213

TerminalSkills/skills

Crawl4AI is an open-source Python web crawler that turns web pages into clean markdown and structured data for LLMs. Use it when asked to scrape a site for a RAG pipeline, convert pages to markdown, render JavaScript pages, extract JSON with CSS selectors or an LLM, or crawl many pages or a whole site with sessions and deep crawling.

Data & Analytics34

Cube

213

TerminalSkills/skills

Cube is an open-source semantic layer that sits between a data warehouse and analytics applications and serves governed metrics over REST, GraphQL and SQL APIs. Use when a user asks to define cubes, measures and dimensions, add pre-aggregations, build a metrics API or embedded analytics, set up multi-tenant row-level security, or query Cube from React or the REST API.

Data & Analytics34

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.