io.github.SK-DEV-AI/websearch

websearch MCP Server

mcp-name: io.github.SK-DEV-AI/websearch

Companion agent skills: SK-DEV-AI/skills · npx skills add SK-DEV-AI/skills

Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.

Tools (20)

• doctor — Run this first. Reports which lanes are usable right now and the exact command to enable the rest. Distinguishes "broken" from "needs optional setup". Never prints key values.
• search — Open research questions. depth=1 snippets, depth≥2 full-page fetch + rerank. synthesize returns a Groq answer with [N] citations.
• fetch — URL to markdown/text. Direct fetch with auto-fallback to a Chromium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (offset). Optional distill (see below). For structured JSON use extract.
• fetch_many — Fetch several URLs in one call. Dedups, caps concurrency, honours robots.txt, shares the cache, returns per-URL success/status/content plus counts. Use when you have 2+ URLs; use fetch for one.
• crawl — Breadth-first site crawl with hard caps (per-page cap, depth cap, robots.txt, dedup, concurrency). Returns each page plus a stop_reason so you know whether the site was exhausted or a cap stopped it. Use for "read this whole docs site".
• extract_links — Every link on a page with anchor text, title and rel. Resolves relative hrefs, drops javascript:/mailto:, dedups. Use before crawl or to find a "next" button.
• extract_table — HTML tables to json, csv or markdown, by selector and index. Reports tables_found so you can pick one.
• download — Save a URL to disk; returns the local path, byte count, sha256 and content type. Writes only inside dest_dir and refuses path traversal. Marks truncated if the byte cap stopped the transfer.
• diff_page — Has this page changed since it was cached? Returns changed/old_hash/new_hash plus added and removed lines. First call for a URL returns had_previous: false.
• screenshot — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
• browser_action — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use fetch. Pass session_id to reuse a persistent tab.
• browser_session — Create, list, inspect or close a persistent browser tab. Without one, every browser_action call gets a fresh tab and closes it, so logins and cookies cannot survive. See below.
• form_fill_submit — Fill many fields, submit, and read the result in one call. Verifies each element exists, reads back each typed value, and refuses to submit if a value did not stick. Reports effect (navigated / content_changed / none) so a submit that silently did nothing is visible rather than assumed.
• paginate — Click a "next" control, collecting a value per page, up to a cap. Stops honestly when the control is missing or has no effect, and reports stop_reason.
• wikipedia — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
• arxiv — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch papers).
• extract — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
• pdf_extract — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
• map_site — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
• ping — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.

What replaced the throwaway scripts

Every one of these exists because an agent otherwise writes a one-off script:

Persistent browser sessions

browser_action opens one fresh tab per call and closes it afterwards. That is right for a single read, but it makes any interactive workflow impossible: no login, no cookies, no scroll position, no multi-step flow across calls.

A session is a named, long-lived tab:

browser_session(action="create", label="login", url="https://site.example/login")
# -> {"session_id": "login-1a2b3c4d", ...}
browser_action(action="type", selector="#email", text="me@example.com", session_id="login-1a2b3c4d")
browser_action(action="click", selector="#submit", session_id="login-1a2b3c4d")
browser_action(action="text", session_id="login-1a2b3c4d")   # same tab, still logged in
browser_session(action="close", session_id="login-1a2b3c4d")

Sessions are opt-in: without session_id, behaviour is exactly as before. They are capped at 8 with least-recently-used eviction, tabs are always closed (including on eviction), and metadata is mirrored to ~/.cache/websearch/sessions.json so list can explain what survived a restart. Sessions need a browser, so they are local-only.

In-tool distillation (optional, off by default)

A fetched page can be 50k characters. The calling model pays for all of it even when it needed one fact. fetch(distill=true) runs a small model inside the tool first and returns a compressed result:

fetch(url="https://example.com/long-report", distill=True,
      distill_question="What are the pricing tiers?")

The result never pretends to be the page:

Below the size threshold (default 4000 chars) distillation is skipped and the raw text is returned, because a summary would cost more than it saves. On any failure the raw text is returned with distill_skipped explaining why - a broken summary is never returned in place of the page.

What works out of the box, and what is optional

Everything keyless works immediately after pip install. No API keys, no browser, no GPU: search (DuckDuckGo, Bing RSS, Wikipedia, arXiv, Stack Exchange, GitHub, HN, grep.app, Google News RSS), fetch, map_site, pdf_extract, screenshot.

Four lanes need optional setup. Run doctor() and it tells you which are live and prints the exact command for the rest:

Enabling the browser lanes:

# any Chromium-family browser: chromium, google-chrome, brave, Helium, edge, vivaldi
chromium --remote-debugging-port=9222 --user-data-dir="$HOME/.mcp-browser"

Then open google.com in that window and sign in once (needed only for Google AI Mode). Point elsewhere with WEBSEARCH_CDP_URL=http://host:port.

# confirm
doctor()                          # per-lane report
search(query="...", google_ai_only=True)   # explicit error + fix list if no browser

The browser lanes are local-only by design. On a server the HTTP lanes are unaffected, so the server is still fully useful — it just cannot query Google AI Mode. Without a browser, google_ai_only=True returns a structured error naming the cause and the fix, never an empty result set.

Setup

Requires Python 3.14+.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp run.example run   # fill in your API keys (run is gitignored — never commit it)
chmod +x run

Point your MCP client at run (stdio):

{ "websearch": { "command": "/path/to/websearch/run" } }

API keys (run)

Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.

Install from PyPI

pip install skdev-websearch-mcp          # core install, small
pip install skdev-websearch-mcp[all]     # + browser/OCR/extract extras (multi-GB)

The websearch-mcp console script speaks MCP over stdio:

{ "websearch": { "command": "websearch-mcp" } }

Docker

docker build -t websearch-mcp .
docker run --rm -i websearch-mcp

Keys are passed at run time, never baked in: -e GROQ_API_KEYS=.... Heavy extras: docker build --build-arg EXTRAS="[all]" -t websearch-mcp .

Notes

• Reranker (optional, local): ranking uses a local gte-reranker-modernbert-base cross-encoder over a Unix socket (RERANKER_MODEL overrides the model). Not bundled with the pip package — set RERANKER_WORKER=/path/to/worker.py to use one. Without it, ranking falls back to blended relevance scores; everything still works. doctor() reports whichever is in effect.
• CDP browser (optional): bot-walled pages escalate to a Chrome DevTools endpoint (WEBSEARCH_CDP_URL, default 127.0.0.1:9222) when available; without it those pages return a terminal verdict instead of hanging.
• Caching: fetch results cache for 1h (cache_ttl=0 forces fresh). No database, no state beyond the cache dir.
• Safety: all user-URL fetches are SSRF-validated (private-IP blocking + redirect-hop revalidation + robots.txt gate).
• Groq model split: synthesis + grounding + extract-llm use openai/gpt-oss-120b; rewrite/classify/expand/decompose use openai/gpt-oss-20b (the previous llama-3.1-8b-instant was retired by Groq and now 404s). Constants GROQ_SMART_MODEL/GROQ_CHEAP_MODEL in config.py.
• Keyless lanes (all advisory-only): OSV.dev vuln vertical + NVD CVE enrichment; grep.app MCP code search (Sourcegraph backup); Bing Web RSS + Bing News RSS (Brave's scrape is rate-limited, GDELT 429s from some networks); Wikipedia vertical; Jina Reader pre-CDP tier; Wayback CDX-history recovery in enrich(); Common Crawl domain-survey assist in map_site.
• Local reranker: a gte-reranker-modernbert-base cross-encoder runs as one shared worker over a Unix socket; either server spawns it on demand and it unloads when idle. Optional — without it results are engine-ranked only.
• Optional keys (graceful keyless fallback): SE_KEY/STACKEXCHANGE_KEY (10k/day SE), S2_KEY/S2_API_KEY (dedicated S2 1 RPS).
• Marginalia: best-effort lane only (shared public key pool, 503s under load); independent-index recall, no dedicated key wired (kontakt@marginalia.nu for one).