Robotstxt AI MCP Server: Install & Setup

Install Robotstxt AI in Claude Code

Claude Code

claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp

mcp.json (Claude Desktop, Cursor, VS Code, Windsurf)

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Yes, most likely. Open source and no paid account or API key is mentioned in the listing. Check the provider for current pricing.

Local. Runs on your machine (stdio). You install it with npx, uvx, Docker or from source.

Run: claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp

bamchi/scrapi-mcp-server

Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.

pcmarts/recruiter-roles-mcp

Search Recruiter Roles: live recruiter jobs, companies, sectors, locations, and market stats.

mouhabmzibra-rgb/mcp-seo-backlinks

Backlinks, Domain Rating & expired-domain drops via seo-backlinks.net (Common Crawl)

rankcontrol/rankcontrol

Plan, write and publish SEO articles on your own site, and track your Google and AI search traffic.

builtbyabs/shopify-geo-audit

Audit any Shopify store for AI search (GEO/AEO) readiness and generate the fixes. No signup.

ethanjyx/openbrand

Extract brand assets (logos, colors, backdrop images, brand name) from any website URL