English Français العربية

polite-crawler

Registre officiel Recherche et web

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Code source Site web Documentation

Installation

Ajoutez ceci à la configuration MCP de votre client (Claude Desktop : claude_desktop_config.json ; Cursor : .cursor/mcp.json ; VS Code et les autres utilisent la même forme « mcpServers »).

NPM package polite-crawler-mcp
Claude Code 
claude mcp add polite-crawler -e 'POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR>' -e 'POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT>' -e 'POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE>' -e 'POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED>' -e 'POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY>' -e 'POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT>' -- npx -y polite-crawler-mcp@0.2.4
Configuration JSON (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "polite-crawler": {
            "command": "npx",
            "args": [
                "-y",
                "polite-crawler-mcp@0.2.4"
            ],
            "env": {
                "POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
                "POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
                "POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
                "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
                "POLITE_CRAWLER_SIGN_KEY": "<POLITE_CRAWLER_SIGN_KEY>",
                "POLITE_CRAWLER_SIGNATURE_AGENT": "<POLITE_CRAWLER_SIGNATURE_AGENT>"
            }
        }
    }
}

Variables d'environnement

  • POLITE_CRAWLER_DIR The folder of the store. The default is ~/.polite-crawler.
  • POLITE_CRAWLER_CONTACT An email address or URL for the From header of each request.
  • POLITE_CRAWLER_ALLOW_PRIVATE Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.
  • POLITE_CRAWLER_ALLOW_USER_PROVIDED Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.
  • POLITE_CRAWLER_SIGN_KEY The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT.
  • POLITE_CRAWLER_SIGNATURE_AGENT Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set.

Documentation

Tirée du README du projet sur GitHub, actualisée chaque semaine. Les liens ouvrent GitHub.

crawler

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler-mcp

What the crawler does for you and for the sites:

  • It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
  • By default, it fetches only public addresses. --allow-private permits loopback and private addresses.
  • The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.
  • The library is polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

License

MIT

Questions fréquentes

Qu'est-ce que polite-crawler ?

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Comment installer polite-crawler ?

Il s'exécute sur votre machine. Dans Claude Code : claude mcp add polite-crawler -e 'POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR>' -e 'POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT>' -e 'POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE>' -e 'POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED>' -e 'POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY>' -e 'POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT>' -- npx -y polite-crawler-mcp@0.2.4. Pour les autres clients, collez la configuration JSON de cette page dans leurs réglages MCP.

Faut-il une clé API pour polite-crawler ?

Son entrée au registre n'indique aucune clé ni secret obligatoire. Certaines fonctions peuvent tout de même demander un compte auprès du service auquel il se connecte.

polite-crawler est-il open source ?

Oui. Son code est sur GitHub sous licence MIT.

Qui maintient polite-crawler ?

Il est publié par mark1russell7. AI.tn le référence sans le maintenir ni le recommander.

Issu du registre MCP officiel. Être listé n'est pas une recommandation : vérifiez le code et les permissions d'un serveur avant de lui donner accès à vos données.