English Français العربية

io.github.sharozdawa/robotstxt-ai

Official registry Search and web

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Source code Documentation

What can you do with io.github.sharozdawa/robotstxt-ai?

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

NPM package robotstxt-ai-mcp
Claude Code 
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp@1.0.1
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "robotstxt-ai": {
            "command": "npx",
            "args": [
                "-y",
                "robotstxt-ai-mcp@1.0.1"
            ]
        }
    }
}

Documentation

From the project's README on GitHub, refreshed weekly. Links open on GitHub.

Contents
  1. Features
  2. MCP Tools
  3. Installation
  4. Tracked Bots
  5. Why robotstxt.ai vs Manual Editing
  6. More Open Source SEO Tools
  7. License

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

Features

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

MCP Tools

ToolDescription
fetch_robotsFetch and analyze a robots.txt from any URL
analyze_robotsAnalyze pasted robots.txt content for AI bot blocking status
generate_robotsGenerate a robots.txt with specified blocked bots and custom rules
list_ai_botsList all known AI bots with user-agents, companies, and descriptions
check_bot_statusCheck if a specific bot is blocked on a given website

Installation

Web App
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev

Open http://localhost:3000 in your browser.

MCP Server — Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}
MCP Server — Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

Tracked Bots

The server knows about 25+ bots including:

  • AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
  • AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
  • Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Why robotstxt.ai vs Manual Editing

Featurerobotstxt.aiManual Editing
Visual toggle UIYesNo
20+ AI bots databaseYesResearch yourself
Analyze existing robots.txtYesNo
MCP ServerYesNo
PriceFreeFree but tedious

More Open Source SEO Tools

ToolDescription
awesome-seo-mcp-serversCurated list of SEO MCP servers and agent skills
indexnow-mcpInstant URL indexing via IndexNow
schema-genSchema.org JSON-LD markup generator
ai-visibilityAI brand visibility tracker

License

MIT

Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert

Frequently asked questions

What is io.github.sharozdawa/robotstxt-ai?

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

How do I install io.github.sharozdawa/robotstxt-ai?

It runs on your machine. In Claude Code: claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp@1.0.1. For other clients, paste the JSON configuration from this page into their MCP settings.

Does io.github.sharozdawa/robotstxt-ai need an API key?

Its registry entry lists no required key or secret. Some features may still need an account with the service it connects to.

Is io.github.sharozdawa/robotstxt-ai open source?

Yes. Its code is on GitHub under the MIT license.

Who maintains io.github.sharozdawa/robotstxt-ai?

It is published by sharozdawa. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.