SearchForge

SearchForge

Open-source web search API and MCP server for LLMs, agents, and RAG.

One local gateway. Intent-aware search. Clean Markdown. REST, MCP, CLI, and TypeScript.

Website · Quick start · Free tools · MCP · API · Design

SearchForge is a free, open-source web search API and MCP server for LLMs, AI agents, and retrieval-augmented generation (RAG) pipelines. It provides a predictable retrieval layer without forcing every project to integrate a paid search vendor. SearchForge routes each query to the right source, isolates provider failures, deduplicates URLs, fuses rankings, and turns public pages into LLM-ready Markdown.

It does not generate answers, hide citations, scrape public SearXNG instances, or send telemetry.

What you get

SearchForge starts with all no-key adapters enabled. auto is the default and routes code, research, and current/community intent to relevant sources while retaining a web fallback. A GitHub token only raises the public API quota, and Brave remains an optional keyed backend. Broad, independent web metasearch is provided by the included SearXNG stack.

Quick start

Try it without cloning

npx --yes --package github:divyanshu-iitian/SearchForge \
  searchforge search "latest open-source agent frameworks"

The first run downloads and builds the package from GitHub. Searches use intent-aware auto routing unless you select a category.

Zero-key local CLI

git clone https://github.com/divyanshu-iitian/SearchForge.git
cd SearchForge
npm install
npm run build

node dist/cli.js search "latest open-source agent frameworks"
node dist/cli.js search "retrieval augmented generation" --category academic
node dist/cli.js search "local LLM tooling" --category community
node dist/cli.js read "https://example.com"
node dist/cli.js doctor

Full web search with private SearXNG

docker compose up --build

curl -s http://localhost:3000/v1/search \
  -H "content-type: application/json" \
  -d '{"query":"open source vector databases","category":"web","limit":5}'

This starts SearchForge on port 3000 and a private, JSON-enabled SearXNG on port 8080. Before exposing the stack, change the SearXNG secret, set SEARCHFORGE_API_KEY, and terminate TLS at a trusted proxy.

Free tools

Search by capability

searchforge search "latest open-source agent frameworks"
searchforge search "browser agent" --category code
searchforge search "semantic reranking" --category academic --json
searchforge search "Show HN search engine" --category community

The default auto category detects code, academic, and current/community signals and queries the matching source families alongside the web fallback. Explicit categories prevent irrelevant providers from being queried. An explicit providers list overrides category routing, which is useful for evaluations.

Read a URL as Markdown

searchforge read "https://example.com/article"

read_url accepts public HTTP(S) URLs only. Credentials, localhost, private IP literals, and non-web protocols are rejected. Responses are size-bounded, timed out, and cached.

Diagnose the whole retrieval path

searchforge doctor

Doctor performs real, bounded probes and reports each provider's access tier, capability, latency, and error. A failed source produces degraded, not a misleading all-or-nothing status.

MCP

SearchForge exposes three stdio tools:

• web_search — routed, citation-ready structured search
• read_url — clean Markdown from a public URL
• search_status — live capability and latency report

{
  "mcpServers": {
    "searchforge": {
      "command": "node",
      "args": ["/absolute/path/to/SearchForge/dist/mcp.js"],
      "env": {
        "SEARCHFORGE_SEARXNG_URL": "http://localhost:8080"
      }
    }
  }
}

The search and status tools return MCP structured content as well as readable text.

Run the MCP server straight from GitHub without a clone:

{
  "mcpServers": {
    "searchforge": {
      "command": "npx",
      "args": [
        "--yes",
        "--package",
        "github:divyanshu-iitian/SearchForge",
        "searchforge-mcp"
      ]
    }
  }
}

SearchForge is also published in the official MCP Registry as io.github.divyanshu-iitian/searchforge. To run the registry-backed OCI image directly from any MCP client that supports a Docker command:

{
  "mcpServers": {
    "searchforge": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "ghcr.io/divyanshu-iitian/searchforge-mcp:0.2.0"
      ]
    }
  }
}

REST API

Search

POST /v1/search
Content-Type: application/json

{
  "query": "open source reranking models",
  "category": "academic",
  "limit": 8,
  "language": "en",
  "freshness": "month",
  "safeSearch": "moderate"
}

{
  "schemaVersion": "1.0",
  "query": "open source reranking models",
  "category": "academic",
  "results": [
    {
      "title": "Example work",
      "url": "https://doi.org/10.0000/example",
      "snippet": "Authors · Publisher · journal-article",
      "source": "crossref",
      "sources": ["crossref"],
      "score": 0.016393
    }
  ],
  "providers": [
    {
      "provider": "crossref",
      "ok": true,
      "latencyMs": 241,
      "resultCount": 8
    }
  ],
  "tookMs": 243,
  "cached": false
}

Read

POST /v1/read
Content-Type: application/json

{"url":"https://example.com/article"}

Other endpoints:

GET /healthz       Process liveness
GET /v1/providers  Configured capabilities and access tiers
GET /v1/doctor     Live dependency health

See the full OpenAPI contract.

TypeScript SDK

import {
  CrossrefProvider,
  GithubProvider,
  JinaReader,
  SearchForge,
} from "searchforge-rag";

const forge = new SearchForge({
  providers: [new GithubProvider(), new CrossrefProvider()],
  reader: new JinaReader(),
  timeoutMs: 8_000,
});

const evidence = await forge.search({
  query: "agentic retrieval",
  category: "academic",
  limit: 10,
});

const page = await forge.read("https://example.com/research");

Until an npm release is published:

npm install github:divyanshu-iitian/SearchForge

Provider details

SearchForge intentionally does not configure public SearXNG instances. They often disable JSON or limit automated traffic; the Docker stack is the stable free path.

How it works

Agent / RAG / MCP client
           |
      validate + route
           |
  +--------+---------+-----------+
  |        |         |           |
 web      code    academic   community       read_url
  |        |         |           |              |
SearXNG  GitHub   Crossref   Hacker News    Jina Reader
Wikipedia
  +--------+---------+-----------+
           |
 normalize -> canonicalize -> deduplicate -> reciprocal-rank fusion
           |
 versioned evidence + provenance + per-source health

Each idempotent provider call has its own abortable timeout. One outage cannot erase healthy results. Tracking parameters are removed before deduplication, and every contributing provider remains in sources.

This capability-first design is inspired by Agent Reach. Agent Reach helps an agent operate many upstream tools directly; SearchForge complements that approach with one stable, embeddable retrieval API for RAG applications.

Configuration

Production boundary

• Set an API key before binding to a public interface.
• Search results and page content are untrusted input; delimit them and apply prompt-injection defenses.
• The built-in cache and rate limiter are process-local. Use shared infrastructure for multiple replicas.
• Provider bodies and credentials are excluded from surfaced errors.
• healthz proves the process is alive; /v1/doctor checks dependencies.

See SECURITY.md, CONTRIBUTING.md, and CHANGELOG.md.

Principles

1. Evidence over generated answers
2. Free and self-hosted paths before vendor lock-in
3. Partial results over total failure
4. Honest capability and quota reporting
5. Stable contracts and explicit provenance
6. No telemetry by default

License

MIT © Divyanshu.

If SearchForge helps your agent, star the repository and share your integration in Discussions.