Install RobotsGate in Claude Code
claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp{
"mcpServers": {
"robotsgate": {
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Frequently asked questions
What does the RobotsGate MCP server do?
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Is the RobotsGate MCP server free?
Yes, most likely. Open source and no paid account or API key is mentioned in the listing. Check the provider for current pricing.
Does RobotsGate run locally or remotely?
Remote. Hosted by the provider. You connect with a URL, nothing to install.
How do I add RobotsGate to Claude Code?
Run: claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp
More search & web scraping MCP servers
View allRobotTraining.jobs
jobs.robottraining/robot-training-jobs
Search open robot training jobs (teleoperation, data collection, AV and drone tests) and their pay.
Robots.txt & AI Crawler Checker
dev.workers.quietworklabs.robots-check/robots-check
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Robutler
ai.robutler/robutler
Build, remix, and run apps with agents on Robutler. Search, delegate, post, chat, manage channels.
RoboTrunk
com.robotrunk/robotrunk
Search independent stores, check recalls and product facts, and prepare checkout at the store.
RocketList
rocketlist-ai/startup-jobs
Search current startup jobs and hiring companies through RocketList's public read-only data.
Robot Skill Hub
Superfast-Robotics/robot-skill-hub
Search robot code and exact-version findings; check evidence before preparing with the Hub CLI.