English Français العربية

io.github.sharozdawa/robotstxt-ai

Registre officiel Recherche et web

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Code source Documentation

Que pouvez-vous faire avec io.github.sharozdawa/robotstxt-ai ?

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

Installation

Ajoutez ceci à la configuration MCP de votre client (Claude Desktop : claude_desktop_config.json ; Cursor : .cursor/mcp.json ; VS Code et les autres utilisent la même forme « mcpServers »).

NPM package robotstxt-ai-mcp
Claude Code 
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp@1.0.1
Configuration JSON (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "robotstxt-ai": {
            "command": "npx",
            "args": [
                "-y",
                "robotstxt-ai-mcp@1.0.1"
            ]
        }
    }
}

Documentation

Tirée du README du projet sur GitHub, actualisée chaque semaine. Les liens ouvrent GitHub.

Sommaire
  1. Features
  2. MCP Tools
  3. Installation
  4. Tracked Bots
  5. Why robotstxt.ai vs Manual Editing
  6. More Open Source SEO Tools
  7. License

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

Features

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

MCP Tools

ToolDescription
fetch_robotsFetch and analyze a robots.txt from any URL
analyze_robotsAnalyze pasted robots.txt content for AI bot blocking status
generate_robotsGenerate a robots.txt with specified blocked bots and custom rules
list_ai_botsList all known AI bots with user-agents, companies, and descriptions
check_bot_statusCheck if a specific bot is blocked on a given website

Installation

Web App
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev

Open http://localhost:3000 in your browser.

MCP Server — Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}
MCP Server — Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

Tracked Bots

The server knows about 25+ bots including:

  • AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
  • AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
  • Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Why robotstxt.ai vs Manual Editing

Featurerobotstxt.aiManual Editing
Visual toggle UIYesNo
20+ AI bots databaseYesResearch yourself
Analyze existing robots.txtYesNo
MCP ServerYesNo
PriceFreeFree but tedious

More Open Source SEO Tools

ToolDescription
awesome-seo-mcp-serversCurated list of SEO MCP servers and agent skills
indexnow-mcpInstant URL indexing via IndexNow
schema-genSchema.org JSON-LD markup generator
ai-visibilityAI brand visibility tracker

License

MIT

Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert

Questions fréquentes

Qu'est-ce que io.github.sharozdawa/robotstxt-ai ?

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Comment installer io.github.sharozdawa/robotstxt-ai ?

Il s'exécute sur votre machine. Dans Claude Code : claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp@1.0.1. Pour les autres clients, collez la configuration JSON de cette page dans leurs réglages MCP.

Faut-il une clé API pour io.github.sharozdawa/robotstxt-ai ?

Son entrée au registre n'indique aucune clé ni secret obligatoire. Certaines fonctions peuvent tout de même demander un compte auprès du service auquel il se connecte.

io.github.sharozdawa/robotstxt-ai est-il open source ?

Oui. Son code est sur GitHub sous licence MIT.

Qui maintient io.github.sharozdawa/robotstxt-ai ?

Il est publié par sharozdawa. AI.tn le référence sans le maintenir ni le recommander.

Issu du registre MCP officiel. Être listé n'est pas une recommandation : vérifiez le code et les permissions d'un serveur avant de lui donner accès à vos données.