English Français العربية

agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)

Official registry Search and web

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

Source code Website Documentation

Install

Add this to your client's MCP configuration (Claude Desktop: claude_desktop_config.json; Cursor: .cursor/mcp.json; VS Code and others use the same "mcpServers" shape).

Remote (streamable-http)
Claude Code 
claude mcp add --transport http agent-web-url-to-llm-ready-markdown-a-po 'https://agent-web.foomworks.workers.dev/mcp'
JSON configuration (Claude Desktop, Cursor, VS Code) 
{
    "mcpServers": {
        "agent-web-url-to-llm-ready-markdown-a-po": {
            "type": "http",
            "url": "https://agent-web.foomworks.workers.dev/mcp"
        }
    }
}

Documentation

From the project's README on GitHub, refreshed weekly. Links open on GitHub.

Contents
  1. agent-web
  2. What it gives an agent
  3. Connect as an MCP server (recommended)
  4. Or call it over HTTP
  5. Charter-clean by design (the bright line)
  6. Limits
  7. Install the AgentSkill
  8. About

agent-web

URL → LLM-ready markdown. A charter-clean, polite, robots-respecting web reader for AI agents — honors robots.txt, identifies honestly, and never bypasses anti-bot/CAPTCHA or paywalls. No account, no API key — free per-call access.

🌐 Live: https://agent-web.foomworks.workers.dev** · 🔌 MCP: **https://agent-web.foomworks.workers.dev/mcp

What it gives an agent

Give it one publicly reachable URL; get back the page as clean, LLM-ready markdown — title, word count, and the text — without parsing raw HTML yourself. Server-rendered (static) HTML pages. Ideal for summarizing an article, extracting docs, or feeding a page into an LLM.

Streamable HTTP, JSON-RPC 2.0:

https://agent-web.foomworks.workers.dev/mcp
ToolCostReturns
read_urlfreefetch a URL → full LLM-ready markdown (title, word count, markdown)
read_url_previewfreethe first ~600 characters of the markdown — a cheap look before the full pull

Or call it over HTTP

BASE=https://agent-web.foomworks.workers.dev
curl -s "$BASE/read?url=https://example.com/&ref=readme"          # full markdown
curl -s "$BASE/read/preview?url=https://example.com/&ref=readme"  # first ~600 chars

Discovery: /.well-known/mcp.json (MCP) · /.well-known/agent-card.json (A2A) · /openapi.json · /policy (Acceptable-Use Policy)

Charter-clean by design (the bright line)

This is a polite reader, not a scraper:

  • Honors the origin's robots.txt for our user-agent — a disallowed path is refused without fetching; an unconfirmable robots.txt (5xx/error) is treated conservatively as disallowed.
  • Identifies honestly with a descriptive user-agent — never spoofs a browser.
  • Read-only GET of the single URL you supply — no crawl, no proxy rotation.
  • Never bypasses anti-bot/CAPTCHA, paywalls, or login walls.
  • SSRF-guarded: private/loopback/link-local/internal hosts are blocked and every redirect hop is re-validated.

You are responsible for your right to fetch the URL you submit.

Limits

  • Static HTML only for now — JavaScript-rendered pages, screenshots, and PDF are a later increment.
  • Body cap ~2 MB, fetch timeout ~12 s.
  • The service holds no keys and never initiates payments.

Install the AgentSkill

SKILL.md is a portable AgentSkill (ClawHub / skills.sh / Claude Skills).

About

agent-web is operated by FOOM — an AI-operated, human-supervised autonomous service. License: MIT.

Frequently asked questions

What is agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)?

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

How do I install agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)?

It runs as a remote server. In Claude Code: claude mcp add --transport http agent-web-url-to-llm-ready-markdown-a-po 'https://agent-web.foomworks.workers.dev/mcp'. For other clients, paste the JSON configuration from this page into their MCP settings.

Does agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free) need an API key?

Its registry entry lists no required key or secret. Some features may still need an account with the service it connects to.

Is agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free) open source?

Yes. Its code is on GitHub under the MIT license.

Who maintains agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)?

It is published by foomworks. AI.tn lists it but does not maintain or endorse it.

From the official MCP Registry. Listing is not an endorsement: review a server's code and permissions before giving it access to your data.