English Français العربية

io.github.sharozdawa/robotstxt-ai

السجل الرسمي البحث والويب

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

الشيفرة المصدرية التوثيق

ماذا يمكنك أن تفعل مع io.github.sharozdawa/robotstxt-ai؟

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

التثبيت

أضف هذا إلى إعدادات MCP في عميلك (Claude Desktop: claude_desktop_config.json، وCursor: ‏.cursor/mcp.json، ويستخدم VS Code وغيره الصيغة نفسها «mcpServers»).

NPM package robotstxt-ai-mcp
Claude Code 
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp@1.0.1
إعداد JSON (Claude Desktop وCursor وVS Code) 
{
    "mcpServers": {
        "robotstxt-ai": {
            "command": "npx",
            "args": [
                "-y",
                "robotstxt-ai-mcp@1.0.1"
            ]
        }
    }
}

التوثيق

مأخوذ من ملف README للمشروع على GitHub ويُحدَّث أسبوعيًا. تفتح الروابط على GitHub.

المحتويات
  1. Features
  2. MCP Tools
  3. Installation
  4. Tracked Bots
  5. Why robotstxt.ai vs Manual Editing
  6. More Open Source SEO Tools
  7. License

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

Features

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

MCP Tools

ToolDescription
fetch_robotsFetch and analyze a robots.txt from any URL
analyze_robotsAnalyze pasted robots.txt content for AI bot blocking status
generate_robotsGenerate a robots.txt with specified blocked bots and custom rules
list_ai_botsList all known AI bots with user-agents, companies, and descriptions
check_bot_statusCheck if a specific bot is blocked on a given website

Installation

Web App
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev

Open http://localhost:3000 in your browser.

MCP Server — Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}
MCP Server — Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

Tracked Bots

The server knows about 25+ bots including:

  • AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
  • AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
  • Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Why robotstxt.ai vs Manual Editing

Featurerobotstxt.aiManual Editing
Visual toggle UIYesNo
20+ AI bots databaseYesResearch yourself
Analyze existing robots.txtYesNo
MCP ServerYesNo
PriceFreeFree but tedious

More Open Source SEO Tools

ToolDescription
awesome-seo-mcp-serversCurated list of SEO MCP servers and agent skills
indexnow-mcpInstant URL indexing via IndexNow
schema-genSchema.org JSON-LD markup generator
ai-visibilityAI brand visibility tracker

License

MIT

Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert

الأسئلة الشائعة

ما هو io.github.sharozdawa/robotstxt-ai؟

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

كيف أثبّت io.github.sharozdawa/robotstxt-ai؟

يعمل على جهازك. في Claude Code: claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp@1.0.1. وفي العملاء الآخرين، الصق إعداد JSON من هذه الصفحة في إعدادات MCP الخاصة بهم.

هل يحتاج io.github.sharozdawa/robotstxt-ai إلى مفتاح API؟

لا يذكر إدراجه في السجل أي مفتاح أو سر إلزامي. قد تتطلب بعض الميزات حسابًا لدى الخدمة التي يتصل بها.

هل io.github.sharozdawa/robotstxt-ai مفتوح المصدر؟

نعم. شيفرته متاحة على GitHub بترخيص MIT.

من يصون io.github.sharozdawa/robotstxt-ai؟

ينشره sharozdawa. يدرجه AI.tn لكنه لا يصونه ولا يوصي به.

من سجل MCP الرسمي. الإدراج ليس توصية: راجع شيفرة الخادم وصلاحياته قبل منحه الوصول إلى بياناتك.