English Français العربية

RobotsGate

Official registry Search and web

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Website

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

Remote (streamable-http)
Claude Code 
claude mcp add --transport http robotsgate 'https://robotsgate.mike-tusa.workers.dev/mcp' --header 'Authorization: <Optional. '\''Bearer <RobotsGate Pro license key>'\'' for higher limits. Works without a key.>'
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "robotsgate": {
            "type": "http",
            "url": "https://robotsgate.mike-tusa.workers.dev/mcp",
            "headers": {
                "Authorization": "<Optional. 'Bearer <RobotsGate Pro license key>' for higher limits. Works without a key.>"
            }
        }
    }
}

Frequently asked questions

What is RobotsGate?

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

How do I install RobotsGate?

It runs as a remote server. In Claude Code: claude mcp add --transport http robotsgate 'https://robotsgate.mike-tusa.workers.dev/mcp' --header 'Authorization: <Optional. '\''Bearer <RobotsGate Pro license key>'\'' for higher limits. Works without a key.>'. For other clients, paste the JSON configuration from this page into their MCP settings.

Does RobotsGate need an API key?

Yes. It expects Authorization, which you set in its configuration.

Who maintains RobotsGate?

It is published by dev.workers.mike-tusa.robotsgate. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.