Scrape-LE MCP Server: Install & Setup

Install Scrape-LE in Claude Code

Claude Code

claude mcp add scrape-le -- npx -y scrape-le-mcp

mcp.json (Claude Desktop, Cursor, VS Code, Windsurf)

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": [
        "-y",
        "scrape-le-mcp"
      ]
    }
  }
}

Analyse robots.txt content and report whether a path may be crawled.

Yes, most likely. Open source and no paid account or API key is mentioned in the listing. Check the provider for current pricing.

Local. Runs on your machine (stdio). You install it with npx, uvx, Docker or from source.

Run: claude mcp add scrape-le -- npx -y scrape-le-mcp

ScrapeGraphAI/Scrapegraph-ai

AI-powered web scraping library that creates scraping pipelines using natural language.- ScrapeGraphAI

yamadashy/repomix

Pack local or remote codebases into AI-friendly files that LLMs and coding agents can read or search

agentskills/agentskills

Search and discover Agent Skills from the skills.sh registry. Powered by HAPI MCP server.

screenpipe/screenpipe

Search your local screen recordings, audio transcripts, and computer activity from screenpipe.

xerj-org/xerj

The search engine for AI agents: cited answers from your local code, docs, specs and logs.

apify/apify-mcp-server

Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡