Install Robots.txt & AI Crawler Checker in Claude Code
claude mcp add --transport http robots-txt-ai-crawler-checker https://robots-check.quietworklabs.workers.dev/mcp{
"mcpServers": {
"robots-txt-ai-crawler-checker": {
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Frequently asked questions
What does the Robots.txt & AI Crawler Checker MCP server do?
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Is the Robots.txt & AI Crawler Checker MCP server free?
The server is free, but you will likely need an account. The server itself is free, but it connects to a service that needs an account or API key, which may have paid tiers. Check the provider for current pricing.
Does Robots.txt & AI Crawler Checker run locally or remotely?
Remote. Hosted by the provider. You connect with a URL, nothing to install.
How do I add Robots.txt & AI Crawler Checker to Claude Code?
Run: claude mcp add --transport http robots-txt-ai-crawler-checker https://robots-check.quietworklabs.workers.dev/mcp
More search & web scraping MCP servers
View allRobotsGate
dev.workers.mike-tusa.robotsgate/robotsgate
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
RoboTrunk
com.robotrunk/robotrunk
Search independent stores, check recalls and product facts, and prepare checkout at the store.
RobotTraining.jobs
jobs.robottraining/robot-training-jobs
Search open robot training jobs (teleoperation, data collection, AV and drone tests) and their pay.
Robot Skill Hub
Superfast-Robotics/robot-skill-hub
Search robot code and exact-version findings; check evidence before preparing with the Hub CLI.
Robutler
ai.robutler/robutler
Build, remix, and run apps with agents on Robutler. Search, delegate, post, chat, manage channels.
RoboHub Robot Catalog
app.robohub/robohub
Vendor-neutral catalog of 300+ commercial robots: search, compare specs, request vendor intros.