TheCrawler MCP Server: Install & Setup

Install TheCrawler in Claude Code

Claude Code

claude mcp add thecrawler -- npx -y thecrawler

mcp.json (Claude Desktop, Cursor, VS Code, Windsurf)

{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler"
      ]
    }
  }
}

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Yes, most likely. Open source and no paid account or API key is mentioned in the listing. Check the provider for current pricing.

Local. Runs on your machine (stdio). You install it with npx, uvx, Docker or from source.

Run: claude mcp add thecrawler -- npx -y thecrawler

ScrapeGraphAI/Scrapegraph-ai

AI-powered web scraping library that creates scraping pipelines using natural language.- ScrapeGraphAI

yamadashy/repomix

Pack local or remote codebases into AI-friendly files that LLMs and coding agents can read or search

agentskills/agentskills

Search and discover Agent Skills from the skills.sh registry. Powered by HAPI MCP server.

screenpipe/screenpipe

Search your local screen recordings, audio transcripts, and computer activity from screenpipe.

xerj-org/xerj

The search engine for AI agents: cited answers from your local code, docs, specs and logs.

apify/apify-mcp-server

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡