English Français العربية

polite-crawler

Official registry Search and web

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Source code Website Documentation

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

NPM package polite-crawler-mcp
Claude Code 
claude mcp add polite-crawler -e 'POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR>' -e 'POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT>' -e 'POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE>' -e 'POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED>' -e 'POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY>' -e 'POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT>' -- npx -y polite-crawler-mcp@0.2.4
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "polite-crawler": {
            "command": "npx",
            "args": [
                "-y",
                "polite-crawler-mcp@0.2.4"
            ],
            "env": {
                "POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
                "POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
                "POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
                "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
                "POLITE_CRAWLER_SIGN_KEY": "<POLITE_CRAWLER_SIGN_KEY>",
                "POLITE_CRAWLER_SIGNATURE_AGENT": "<POLITE_CRAWLER_SIGNATURE_AGENT>"
            }
        }
    }
}

Environment variables

  • POLITE_CRAWLER_DIR The folder of the store. The default is ~/.polite-crawler.
  • POLITE_CRAWLER_CONTACT An email address or URL for the From header of each request.
  • POLITE_CRAWLER_ALLOW_PRIVATE Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.
  • POLITE_CRAWLER_ALLOW_USER_PROVIDED Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.
  • POLITE_CRAWLER_SIGN_KEY The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT.
  • POLITE_CRAWLER_SIGNATURE_AGENT Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set.

Documentation

From the project's README on GitHub, refreshed weekly. Links open on GitHub.

crawler

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler-mcp

What the crawler does for you and for the sites:

  • It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
  • By default, it fetches only public addresses. --allow-private permits loopback and private addresses.
  • The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.
  • The library is polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

License

MIT

Frequently asked questions

What is polite-crawler?

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

How do I install polite-crawler?

It runs on your machine. In Claude Code: claude mcp add polite-crawler -e 'POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR>' -e 'POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT>' -e 'POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE>' -e 'POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED>' -e 'POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY>' -e 'POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT>' -- npx -y polite-crawler-mcp@0.2.4. For other clients, paste the JSON configuration from this page into their MCP settings.

Does polite-crawler need an API key?

Its registry entry lists no required key or secret. Some features may still need an account with the service it connects to.

Is polite-crawler open source?

Yes. Its code is on GitHub under the MIT license.

Who maintains polite-crawler?

It is published by mark1russell7. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.