English Français العربية

AI Crawler Checker — Robots.txt Policy for Six Bots

Official registry Search and web

Check declared robots.txt permissions for six AI crawlers at URL paths

Website

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

Remote (streamable-http)
Claude Code 
claude mcp add --transport http ai-crawler-checker-robots-txt-policy-for 'https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check' --header 'Authorization: <Optional: '\''Bearer <your Apify API token>'\''. MCP clients that support OAuth can sign in to Apify instead. Runs are billed to your Apify account at the pay-per-event prices on the Actor'\''s store page.>'
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "ai-crawler-checker-robots-txt-policy-for": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
            "headers": {
                "Authorization": "<Optional: 'Bearer <your Apify API token>'. MCP clients that support OAuth can sign in to Apify instead. Runs are billed to your Apify account at the pay-per-event prices on the Actor's store page.>"
            }
        }
    }
}

Frequently asked questions

What is AI Crawler Checker — Robots.txt Policy for Six Bots?

Check declared robots.txt permissions for six AI crawlers at URL paths

How do I install AI Crawler Checker — Robots.txt Policy for Six Bots?

It runs as a remote server. In Claude Code: claude mcp add --transport http ai-crawler-checker-robots-txt-policy-for 'https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check' --header 'Authorization: <Optional: '\''Bearer <your Apify API token>'\''. MCP clients that support OAuth can sign in to Apify instead. Runs are billed to your Apify account at the pay-per-event prices on the Actor'\''s store page.>'. For other clients, paste the JSON configuration from this page into their MCP settings.

Does AI Crawler Checker — Robots.txt Policy for Six Bots need an API key?

Yes. It expects Authorization, which you set in its configuration.

Who maintains AI Crawler Checker — Robots.txt Policy for Six Bots?

It is published by kriton-dev. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.