English Français العربية

agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)

السجل الرسمي البحث والويب

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

الشيفرة المصدرية الموقع التوثيق

التثبيت

أضف هذا إلى إعدادات MCP في عميلك (Claude Desktop: claude_desktop_config.json، وCursor: ‏.cursor/mcp.json، ويستخدم VS Code وغيره الصيغة نفسها «mcpServers»).

Remote (streamable-http)
Claude Code 
claude mcp add --transport http agent-web-url-to-llm-ready-markdown-a-po 'https://agent-web.foomworks.workers.dev/mcp'
إعداد JSON (Claude Desktop وCursor وVS Code) 
{
    "mcpServers": {
        "agent-web-url-to-llm-ready-markdown-a-po": {
            "type": "http",
            "url": "https://agent-web.foomworks.workers.dev/mcp"
        }
    }
}

التوثيق

مأخوذ من ملف README للمشروع على GitHub ويُحدَّث أسبوعيًا. تفتح الروابط على GitHub.

المحتويات
  1. agent-web
  2. What it gives an agent
  3. Connect as an MCP server (recommended)
  4. Or call it over HTTP
  5. Charter-clean by design (the bright line)
  6. Limits
  7. Install the AgentSkill
  8. About

agent-web

URL → LLM-ready markdown. A charter-clean, polite, robots-respecting web reader for AI agents — honors robots.txt, identifies honestly, and never bypasses anti-bot/CAPTCHA or paywalls. No account, no API key — free per-call access.

🌐 Live: https://agent-web.foomworks.workers.dev** · 🔌 MCP: **https://agent-web.foomworks.workers.dev/mcp

What it gives an agent

Give it one publicly reachable URL; get back the page as clean, LLM-ready markdown — title, word count, and the text — without parsing raw HTML yourself. Server-rendered (static) HTML pages. Ideal for summarizing an article, extracting docs, or feeding a page into an LLM.

Streamable HTTP, JSON-RPC 2.0:

https://agent-web.foomworks.workers.dev/mcp
ToolCostReturns
read_urlfreefetch a URL → full LLM-ready markdown (title, word count, markdown)
read_url_previewfreethe first ~600 characters of the markdown — a cheap look before the full pull

Or call it over HTTP

BASE=https://agent-web.foomworks.workers.dev
curl -s "$BASE/read?url=https://example.com/&ref=readme"          # full markdown
curl -s "$BASE/read/preview?url=https://example.com/&ref=readme"  # first ~600 chars

Discovery: /.well-known/mcp.json (MCP) · /.well-known/agent-card.json (A2A) · /openapi.json · /policy (Acceptable-Use Policy)

Charter-clean by design (the bright line)

This is a polite reader, not a scraper:

  • Honors the origin's robots.txt for our user-agent — a disallowed path is refused without fetching; an unconfirmable robots.txt (5xx/error) is treated conservatively as disallowed.
  • Identifies honestly with a descriptive user-agent — never spoofs a browser.
  • Read-only GET of the single URL you supply — no crawl, no proxy rotation.
  • Never bypasses anti-bot/CAPTCHA, paywalls, or login walls.
  • SSRF-guarded: private/loopback/link-local/internal hosts are blocked and every redirect hop is re-validated.

You are responsible for your right to fetch the URL you submit.

Limits

  • Static HTML only for now — JavaScript-rendered pages, screenshots, and PDF are a later increment.
  • Body cap ~2 MB, fetch timeout ~12 s.
  • The service holds no keys and never initiates payments.

Install the AgentSkill

SKILL.md is a portable AgentSkill (ClawHub / skills.sh / Claude Skills).

About

agent-web is operated by FOOM — an AI-operated, human-supervised autonomous service. License: MIT.

الأسئلة الشائعة

ما هو agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)؟

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

كيف أثبّت agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)؟

يعمل كخادم بعيد. في Claude Code: claude mcp add --transport http agent-web-url-to-llm-ready-markdown-a-po 'https://agent-web.foomworks.workers.dev/mcp'. وفي العملاء الآخرين، الصق إعداد JSON من هذه الصفحة في إعدادات MCP الخاصة بهم.

هل يحتاج agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free) إلى مفتاح API؟

لا يذكر إدراجه في السجل أي مفتاح أو سر إلزامي. قد تتطلب بعض الميزات حسابًا لدى الخدمة التي يتصل بها.

هل agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free) مفتوح المصدر؟

نعم. شيفرته متاحة على GitHub بترخيص MIT.

من يصون agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)؟

ينشره foomworks. يدرجه AI.tn لكنه لا يصونه ولا يوصي به.

من سجل MCP الرسمي. الإدراج ليس توصية: راجع شيفرة الخادم وصلاحياته قبل منحه الوصول إلى بياناتك.