English Français العربية

AI-First Scraper

السجل الرسمي البحث والويب

Three MCP tools: fetch_page, fetch_pages_batch, search_web. Ad-free Markdown for AI agents.

الشيفرة المصدرية التوثيق

التثبيت

أضف هذا إلى إعدادات MCP في عميلك (Claude Desktop: claude_desktop_config.json، وCursor: ‏.cursor/mcp.json، ويستخدم VS Code وغيره الصيغة نفسها «mcpServers»).

PYPI package ai-first-scraper-mcp
Claude Code 
claude mcp add ai-first-scraper -e 'SCRAPER_URL=<SCRAPER_URL>' -e 'SEARCH_URL=<SEARCH_URL>' -e 'AFS_TIMEOUT=<AFS_TIMEOUT>' -- uvx ai-first-scraper-mcp==1.0.2
إعداد JSON (Claude Desktop وCursor وVS Code) 
{
    "mcpServers": {
        "ai-first-scraper": {
            "command": "uvx",
            "args": [
                "ai-first-scraper-mcp==1.0.2"
            ],
            "env": {
                "SCRAPER_URL": "<SCRAPER_URL>",
                "SEARCH_URL": "<SEARCH_URL>",
                "AFS_TIMEOUT": "<AFS_TIMEOUT>"
            }
        }
    }
}

متغيرات البيئة

  • SCRAPER_URL URL of the ai-first-scraper instance to use. Default: https://ai-first-scraper.onrender.com
  • SEARCH_URL URL of the ai-first-search instance to use. Default: https://ai-first-search.onrender.com
  • AFS_TIMEOUT Upstream HTTP timeout in seconds. Default: 45

التوثيق

مأخوذ من ملف README للمشروع على GitHub ويُحدَّث أسبوعيًا. تفتح الروابط على GitHub.

المحتويات
  1. ai-first-scraper-mcp
  2. What it does
  3. Install
  4. Where the config file lives
  5. Point at your own backend (optional)
  6. Verify it works
  7. Companion projects
  8. Develop locally
  9. About the author
  10. License

ai-first-scraper-mcp

Plug Claude Desktop, Cursor, or Cline straight into an ad-free web scraper + search engine. Three tools, one line of config.

[]() []() []()

What it does

Adds three tools to any MCP-compatible agent:

ToolWhat it does
fetch_pageFetch one URL → return clean Markdown (HTML or PDF).
fetch_pages_batchFetch up to 25 URLs in parallel → return Markdown for each.
search_webRun a web search and return the top-k result pages already converted to Markdown.

No more "the model called curl and then tried to parse 80kB of ad HTML." Your agent receives clean Markdown ready to reason about.

Backed by the ai-first-scraper and ai-first-search APIs.

Install

Fastest — uvx (no install, runs from PyPI on demand)
// claude_desktop_config.json  /  cline_mcp_settings.json  /  ~/.cursor/mcp.json
{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "uvx",
      "args": ["ai-first-scraper-mcp"]
    }
  }
}

Restart your client (Claude Desktop / Cursor / Cline). The three tools above will appear automatically.

Alternative — pip install
pip install ai-first-scraper-mcp
{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "ai-first-scraper-mcp"
    }
  }
}

Where the config file lives

ClientConfig path
Claude Desktop (macOS)~/Library/Application Support/Claude/claude_desktop_config.json
Claude Desktop (Windows)%APPDATA%\Claude\claude_desktop_config.json
Cursor~/.cursor/mcp.json
Cline (VS Code)~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json

Point at your own backend (optional)

By default this server calls the public ai-first-scraper.onrender.com and ai-first-search.onrender.com instances. If you want to self-host, set env vars in your MCP config:

{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "uvx",
      "args": ["ai-first-scraper-mcp"],
      "env": {
        "SCRAPER_URL": "https://your-scraper.example.com",
        "SEARCH_URL":  "https://your-search.example.com",
        "AFS_TIMEOUT": "60"
      }
    }
  }
}

Verify it works

Open your MCP client and ask the agent:

"Use the search_web tool to find the top 3 recent articles about MCP and summarize them in 5 bullets each."

You should see the agent call search_web, get back Markdown for each result, and produce the summary without ever touching raw HTML.

Companion projects

  • ai-first-scraper — the per-URL Markdown cleaner this MCP server fans out to.
  • ai-first-search — search → scrape → markdown pipeline.
  • mcp-rec — record & replay any MCP server's traffic for tests and bug reports.
  • llm-cache-proxy — local cache for OpenAI/Anthropic API calls.
  • promptlocker — lockfile for prompts.
  • context-diff — see what blew up your Claude Code context window.
  • agentwatch — overlay for browser AI agents.

Develop locally

git clone https://github.com/yubinkim444/ai-first-scraper-mcp.git
cd ai-first-scraper-mcp

uv sync                    # or: pip install -e .
ai-first-scraper-mcp       # speaks MCP over stdio

To test against a local client, point its MCP config at the same command.

About the author

Built by yubinkim444, who also makes Kay's Records — an app for iOS and Android.

If this project saved you time, giving the app a try is the nicest way to say thanks.

License

MIT © yubinkim444

الأسئلة الشائعة

ما هو AI-First Scraper؟

Three MCP tools: fetch_page, fetch_pages_batch, search_web. Ad-free Markdown for AI agents.

كيف أثبّت AI-First Scraper؟

يعمل على جهازك. في Claude Code: claude mcp add ai-first-scraper -e 'SCRAPER_URL=<SCRAPER_URL>' -e 'SEARCH_URL=<SEARCH_URL>' -e 'AFS_TIMEOUT=<AFS_TIMEOUT>' -- uvx ai-first-scraper-mcp==1.0.2. وفي العملاء الآخرين، الصق إعداد JSON من هذه الصفحة في إعدادات MCP الخاصة بهم.

هل يحتاج AI-First Scraper إلى مفتاح API؟

لا يذكر إدراجه في السجل أي مفتاح أو سر إلزامي. قد تتطلب بعض الميزات حسابًا لدى الخدمة التي يتصل بها.

هل AI-First Scraper مفتوح المصدر؟

نعم. شيفرته متاحة على GitHub بترخيص MIT.

من يصون AI-First Scraper؟

ينشره yubinkim444. يدرجه AI.tn لكنه لا يصونه ولا يوصي به.

من سجل MCP الرسمي. الإدراج ليس توصية: راجع شيفرة الخادم وصلاحياته قبل منحه الوصول إلى بياناتك.