English Français العربية

polite-crawler

السجل الرسمي البحث والويب

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

الشيفرة المصدرية الموقع التوثيق

التثبيت

أضف هذا إلى إعدادات MCP في عميلك (Claude Desktop: claude_desktop_config.json، وCursor: ‏.cursor/mcp.json، ويستخدم VS Code وغيره الصيغة نفسها «mcpServers»).

NPM package polite-crawler-mcp
Claude Code 
claude mcp add polite-crawler -e 'POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR>' -e 'POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT>' -e 'POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE>' -e 'POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED>' -e 'POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY>' -e 'POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT>' -- npx -y polite-crawler-mcp@0.2.4
إعداد JSON (Claude Desktop وCursor وVS Code) 
{
    "mcpServers": {
        "polite-crawler": {
            "command": "npx",
            "args": [
                "-y",
                "polite-crawler-mcp@0.2.4"
            ],
            "env": {
                "POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
                "POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
                "POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
                "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>",
                "POLITE_CRAWLER_SIGN_KEY": "<POLITE_CRAWLER_SIGN_KEY>",
                "POLITE_CRAWLER_SIGNATURE_AGENT": "<POLITE_CRAWLER_SIGNATURE_AGENT>"
            }
        }
    }
}

متغيرات البيئة

  • POLITE_CRAWLER_DIR The folder of the store. The default is ~/.polite-crawler.
  • POLITE_CRAWLER_CONTACT An email address or URL for the From header of each request.
  • POLITE_CRAWLER_ALLOW_PRIVATE Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.
  • POLITE_CRAWLER_ALLOW_USER_PROVIDED Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.
  • POLITE_CRAWLER_SIGN_KEY The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT.
  • POLITE_CRAWLER_SIGNATURE_AGENT Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set.

التوثيق

مأخوذ من ملف README للمشروع على GitHub ويُحدَّث أسبوعيًا. تفتح الروابط على GitHub.

crawler

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler-mcp

What the crawler does for you and for the sites:

  • It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
  • By default, it fetches only public addresses. --allow-private permits loopback and private addresses.
  • The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.
  • The library is polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

License

MIT

الأسئلة الشائعة

ما هو polite-crawler؟

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

كيف أثبّت polite-crawler؟

يعمل على جهازك. في Claude Code: claude mcp add polite-crawler -e 'POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR>' -e 'POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT>' -e 'POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE>' -e 'POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED>' -e 'POLITE_CRAWLER_SIGN_KEY=<POLITE_CRAWLER_SIGN_KEY>' -e 'POLITE_CRAWLER_SIGNATURE_AGENT=<POLITE_CRAWLER_SIGNATURE_AGENT>' -- npx -y polite-crawler-mcp@0.2.4. وفي العملاء الآخرين، الصق إعداد JSON من هذه الصفحة في إعدادات MCP الخاصة بهم.

هل يحتاج polite-crawler إلى مفتاح API؟

لا يذكر إدراجه في السجل أي مفتاح أو سر إلزامي. قد تتطلب بعض الميزات حسابًا لدى الخدمة التي يتصل بها.

هل polite-crawler مفتوح المصدر؟

نعم. شيفرته متاحة على GitHub بترخيص MIT.

من يصون polite-crawler؟

ينشره mark1russell7. يدرجه AI.tn لكنه لا يصونه ولا يوصي به.

من سجل MCP الرسمي. الإدراج ليس توصية: راجع شيفرة الخادم وصلاحياته قبل منحه الوصول إلى بياناتك.